Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,14 @@ updates:
- "*"
cooldown:
default-days: 30

- package-ecosystem: pre-commit
directory: /
schedule:
interval: monthly
groups:
pre-commit:
patterns:
- "*"
cooldown:
default-days: 30
8 changes: 4 additions & 4 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,9 @@ jobs:
name: "CDK Operations"
runs-on: ubuntu-latest
timeout-minutes: 60

environment: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.deployment_environment || 'synthtest' }}

env:
AUTHOR: ${{ github.actor }}
CERTIFICATE_ARN: ${{ vars.CERTIFICATE_ARN }}
Expand Down Expand Up @@ -55,7 +55,7 @@ jobs:
USER_STAC_STAC_API_CUSTOM_DOMAIN_NAME: ${{ vars.USER_STAC_STAC_API_CUSTOM_DOMAIN_NAME }}
USER_STAC_TITILER_PGSTAC_API_CUSTOM_DOMAIN_NAME: ${{ vars.USER_STAC_TITILER_PGSTAC_API_CUSTOM_DOMAIN_NAME }}
WEB_ACL_ARN: ${{ vars.WEB_ACL_ARN }}

steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
Expand Down Expand Up @@ -89,7 +89,7 @@ jobs:
run: |
echo "INGESTOR_DATA_ACCESS_ROLE_ARN=${{ steps.import-stacks-vars-to-output.outputs.INGESTOR_DATA_ACCESS_ROLE_ARN }}" >> $GITHUB_ENV
echo "JWKS_URL=${{ steps.import-stacks-vars-to-output.outputs.JWKS_URL }}" >> $GITHUB_ENV

- name: Set fallback dynamic variables for synthtest
if: github.event_name != 'workflow_dispatch'
run: |
Expand Down
47 changes: 47 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Configuration file for pre-commit (https://pre-commit.com/).
# Please run `pre-commit run --all-files` when adding or changing entries.

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v6.0.0
hooks:
- id: check-yaml
- id: trailing-whitespace

- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.16.4
hooks:
- id: ruff-check
args: [--fix]
- id: ruff-format
types_or: [python, pyi, pyproject]

- repo: https://github.com/DavidAnson/markdownlint-cli2
rev: v0.23.2
hooks:
- id: markdownlint-cli2
args:
- "--fix"
- "--config"
- "pyproject.toml"
- "--configPointer"
- "/tool/markdownlint-cli2"

- repo: https://github.com/astral-sh/uv-pre-commit
rev: 0.12.5
hooks:
- id: uv-lock

- repo: https://github.com/tsvikas/sync-with-uv
rev: v0.6.0
hooks:
- id: sync-with-uv

- repo: https://github.com/pre-commit/mirrors-mypy
rev: v2.3.1
hooks:
- id: mypy
additional_dependencies:
- types-requests
- pydantic==2.13.5 # sync-with-uv
- pydantic-settings==2.15.0 # sync-with-uv
28 changes: 11 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,10 @@

[![Tests Status](https://github.com/MAAP-Project/maap-eoapi/actions/workflows/tests.yml/badge.svg)]((https://github.com/MAAP-Project/maap-eoapi/actions?query=workflow:tests))


## Overview

This repository contains the AWS CDK code (written in Python) used to deploy the MAAP project eoapi infrastructure. It is based on the [eoapi-template example](https://github.com/developmentseed/eoapi-template). For the MAAP use case, we use a subset of the eoapi CDK constructs to define a database, an ingestion API, a STAC API, a raster API (i.e a tiling API) and a pgbouncer instance to manage connections to the database. Here, we deploy all these components into a custom VPC.


## Automated Deployment

Deployment happens through a github workflow manually triggered and defined in `.github/workflows/deploy.yaml`.
Expand Down Expand Up @@ -82,40 +80,36 @@ For a transaction-enabled internal deployment, verify:
- authenticated writes succeed
- item write routes are absent from the contract and return `404` or `405` rather than exposing item transaction behavior

## Networking and accessibility of the database

## Networking and accessibility of the database.

Because of security requirements, the networking set up imposes the following constraints :

- For security reasons, the database is in a _private_ subnet of the VPC. As such, only instances running inside of the same VPC can access the database. This means that, for example, even if a user has the password and her IP is allowed inbound connections to the database, access will _not_ be allowed.
Because of security requirements, the networking set up imposes the following constraints :

This has three consequences :
- For security reasons, the database is in a _private_ subnet of the VPC. As such, only instances running inside of the same VPC can access the database. This means that, for example, even if a user has the password and her IP is allowed inbound connections to the database, access will _not_ be allowed.

1. The APIs that need access to the database (the STAC API, the tiling API, the ingestion API) need to be deployed in that same VPC.
2. In addition, because these APIs _also_ sometimes need access to the internet, a NAT gateway must in addition be deployed in that VPC.
3. For direct, administrative connections to the database, one _must_ go through an instance placed in the same VPC as the database.
This has three consequences :

1. The APIs that need access to the database (the STAC API, the tiling API, the ingestion API) need to be deployed in that same VPC.
2. In addition, because these APIs _also_ sometimes need access to the internet, a NAT gateway must in addition be deployed in that VPC.
3. For direct, administrative connections to the database, one _must_ go through an instance placed in the same VPC as the database.

## Ingestion

The term "ingestion" refers to the process of cataloging data in the STAC catalog associated with this deployment.

The term "ingestion" refers to the process of cataloging data in the STAC catalog associated with this deployment.

### Direct ingestion

For a small record ingestion (for example a collection record or just one item), one can directly connect to the database and perform loading. This can be done using the `pypgstac` library. For example, to load an item stored locally in `test_item.json`, with `pypgstac` installed, you can run the following command :
For a small record ingestion (for example a collection record or just one item), one can directly connect to the database and perform loading. This can be done using the `pypgstac` library. For example, to load an item stored locally in `test_item.json`, with `pypgstac` installed, you can run the following command :

```
```shell
pypgstac load --table items test_item.json
```

or for a collection

```
```shell
pypgstac load --table collections test_collection.json
```


### Indirect ingestion through the ingestion pipeline deployment

For larger scale ingestions, in MAAP we rely on [a fork of the stactools-pipelines repository](https://github.com/MAAP-Project/stactools-pipelines/tree/non-standard-inventory). If you want to ingest a collection in MAAP using this tool, you should develop a 'pipeline'. Details of this procedure can be found in the linked repository. You can follow an example that [was developed for maap here](https://github.com/MAAP-Project/stactools-pipelines/tree/non-standard-inventory/stactools_pipelines/pipelines/nisar-sim).
2 changes: 1 addition & 1 deletion app.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@
titiler_pgstac_config=config.user_titiler_pgstac(),
add_stactools_item_generator=False,
**(
{"dps_stac_item_gen_config": dps_stac_item_gen_config}
{"dps_stac_item_gen_config": dps_stac_item_gen_config} # type: ignore [arg-type]
if dps_stac_item_gen_config
else {}
),
Expand Down
83 changes: 52 additions & 31 deletions cdk/config.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
from __future__ import annotations

from typing import Literal, Optional
from dataclasses import dataclass
from typing import Literal

from aws_cdk import aws_ec2 as ec2
from pydantic import (
AliasChoices,
Expand All @@ -25,22 +26,25 @@ class PgStacDbConfig:
class TitilerPgstacConfig:
buckets_path: str
data_access_role_arn: str
mosaic_host: Optional[str] = None
custom_domain_name: Optional[str] = None
mosaic_host: str | None = None
custom_domain_name: str | None = None


TransactionAuthMode = Literal["basic", "jwt"]


@dataclass
class CollectionTransactionsConfig:
auth_mode: Literal["basic", "jwt"]
auth_secret_arn: Optional[str] = None
auth_mode: TransactionAuthMode
auth_secret_arn: str | None = None


@dataclass
class StacApiConfig:
custom_domain_name: Optional[str] = None
integration_api_arn: Optional[str] = None
transactions: Optional[CollectionTransactionsConfig] = None
catalogs: Optional["StacCatalogsConfig"] = None
custom_domain_name: str | None = None
integration_api_arn: str | None = None
transactions: CollectionTransactionsConfig | None = None
catalogs: StacCatalogsConfig | None = None


@dataclass
Expand All @@ -55,21 +59,21 @@ class IngestorConfig:
jwks_url: str
data_access_role_arn: str
user_data_path: str
domain_name: Optional[str] = None
domain_name: str | None = None


@dataclass
class DpsStacItemGenConfig:
item_gen_role_arn: str
inbound_topic_arns: Optional[list[str]] = None
user_stac_collection_id_registry: Optional[dict[str, list[str]]] = None
inbound_topic_arns: list[str] | None = None
user_stac_collection_id_registry: dict[str, list[str]] | None = None


@dataclass
class StacCatalogsConfig:
enabled: bool
hide_alternate_parents: Optional[bool] = None
transactions: Optional[CollectionTransactionsConfig] = None
hide_alternate_parents: bool | None = None
transactions: CollectionTransactionsConfig | None = None


class Config(BaseSettings):
Expand Down Expand Up @@ -97,31 +101,48 @@ class Config(BaseSettings):

# --- Optional ---
version: str = "0.1.1"
certificate_arn: Optional[str] = None
ingestor_domain_name: Optional[str] = None
certificate_arn: str | None = None
ingestor_domain_name: str | None = None
# env var is TITILER_PGSTAC_API_CUSTOM_DOMAIN_NAME (no underscore between pg/stac)
titiler_pg_stac_api_custom_domain_name: Optional[str] = Field(
titiler_pg_stac_api_custom_domain_name: str | None = Field(
None,
validation_alias=AliasChoices(
"titiler_pgstac_api_custom_domain_name",
"titiler_pg_stac_api_custom_domain_name",
),
)
user_stac_item_gen_role_arn: Optional[str] = None
user_stac_stac_api_custom_domain_name: Optional[str] = None
user_stac_titiler_pgstac_api_custom_domain_name: Optional[str] = None
user_stac_inbound_topic_arns: Optional[list[str]] = None
user_stac_collection_id_registry: Optional[dict[str, list[str]]] = None
user_stac_item_gen_role_arn: str | None = None
user_stac_stac_api_custom_domain_name: str | None = None
user_stac_titiler_pgstac_api_custom_domain_name: str | None = None
user_stac_inbound_topic_arns: list[str] | None = None
user_stac_collection_id_registry: dict[str, list[str]] | None = None

# --- Collection transactions env fields ---
user_stac_collection_transactions_auth_mode: Optional[str] = None
user_stac_collection_transactions_auth_secret_arn: Optional[str] = None
user_stac_collection_transactions_auth_mode: TransactionAuthMode | None = None
user_stac_collection_transactions_auth_secret_arn: str | None = None

# --- Catalog env fields ---
user_stac_catalogs_enabled: bool = True
user_stac_catalogs_hide_alternate_parents: Optional[bool] = None
user_stac_catalog_transactions_auth_mode: Optional[str] = None
user_stac_catalog_transactions_auth_secret_arn: Optional[str] = None
user_stac_catalogs_hide_alternate_parents: bool | None = None
user_stac_catalog_transactions_auth_mode: TransactionAuthMode | None = None
user_stac_catalog_transactions_auth_secret_arn: str | None = None

@field_validator(
"user_stac_collection_transactions_auth_mode",
"user_stac_catalog_transactions_auth_mode",
mode="before",
)
@classmethod
def parse_auth_mode(cls, v: object) -> str | None:
if v is None:
return None
if isinstance(v, str):
normalized = v.strip().lower()
if normalized in {"basic", "jwt"}:
return normalized
raise ValueError(
f"Unsupported auth mode: {v!r}. Expected one of: 'basic', 'jwt'."
)

@field_validator("db_instance_type", mode="before")
@classmethod
Expand All @@ -138,7 +159,7 @@ def parse_instance_type(cls, v: object) -> ec2.InstanceType:
mode="before",
)
@classmethod
def parse_optional_bool_env(cls, v: object) -> Optional[bool]:
def parse_optional_bool_env(cls, v: object) -> bool | None:
if v is None:
return None
if isinstance(v, bool):
Expand Down Expand Up @@ -216,7 +237,7 @@ def validate_collection_transactions(self) -> Config:
@property
def user_stac_collection_transactions(
self,
) -> Optional[CollectionTransactionsConfig]:
) -> CollectionTransactionsConfig | None:
if self.user_stac_collection_transactions_auth_mode is None:
return None
return CollectionTransactionsConfig(
Expand All @@ -227,7 +248,7 @@ def user_stac_collection_transactions(
@computed_field # type: ignore[prop-decorator]
@property
def user_stac_catalogs(self) -> StacCatalogsConfig:
catalog_transactions: Optional[CollectionTransactionsConfig] = None
catalog_transactions: CollectionTransactionsConfig | None = None
if self.user_stac_catalog_transactions_auth_mode is not None:
catalog_transactions = CollectionTransactionsConfig(
auth_mode=self.user_stac_catalog_transactions_auth_mode,
Expand Down Expand Up @@ -301,7 +322,7 @@ def ingestor(self) -> IngestorConfig:
user_data_path="./userdata.yaml",
)

def dps_stac_item_gen(self) -> Optional[DpsStacItemGenConfig]:
def dps_stac_item_gen(self) -> DpsStacItemGenConfig | None:
if not self.user_stac_item_gen_role_arn:
return None
return DpsStacItemGenConfig(
Expand Down
2 changes: 1 addition & 1 deletion cdk/constructs/DpsStacItemGenerator/runtime/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,6 @@ COPY runtime/pyproject.toml pyproject.toml
COPY runtime/src/dps_stac_item_generator/ dps_stac_item_generator/

RUN uv export --no-dev --no-editable -o requirements.txt && \
uv pip install --target /asset -r requirements.txt
uv pip install --target /asset -r requirements.txt

CMD ["dps_stac_item_generator.handler.handler"]
Loading