Skip to content

feat: Open Data Catalog explorer with server-side clip#94

Open
prbatero wants to merge 4 commits into
mainfrom
feat-data-catalog-with-img-prep
Open

feat: Open Data Catalog explorer with server-side clip#94
prbatero wants to merge 4 commits into
mainfrom
feat-data-catalog-with-img-prep

Conversation

@prbatero

Copy link
Copy Markdown
Collaborator

Add an Open Data Catalog side panel to the Create Image Layer page so analysts can browse open disaster imagery (Vantor/Maxar + Planet Open Data), preview it, and build an image layer without hunting for COG URLs.

Catalog UI (ui/src/Components/OpenDataCatalog/):

  • Dynamic discovery + merge of every event across the Vantor S3 STAC and Planet Source-Cooperative STAC catalogs; framework-free module with a discoverEvents()/fetchEventCatalog() contract (swap-to-backend seam).
  • Azure Maps footprints + TiTiler COG preview (works with no Azure Maps key); preview tile loading indicator.
  • Add a scene to pre/post in one click with source-type + capture-date auto-fill; buttons are phase-scoped (pre->Pre, post->Post).
  • Two-way map<->list selection: click a footprint to scroll/highlight/expand the list row; footprints stay clickable during preview.

Server-side clip AOI:

  • Draw a box that sets a layer-level clipBbox ([w,s,e,n] EPSG:4326); imagery prep clips the pre/post mosaics via gdalwarp -te/-te_srs (GDAL reprojects the bounds). The catalog filters scenes overlapping the AOI and badges scenes that fully cover it, so pre/post share one AOI.
  • Model ImageLayer.clipBbox + validate_clip_bbox; PutLayer validates/carries it; threaded through the imagery-prep config -> ImageryWorkflow -> mosaic_imagery. Verified end-to-end (output bounds match the drawn box).

Supporting fixes:

  • Allow data.source.coop on the imagery allowlist (UI + hastelib) and route non-S3/Blob allowlisted hosts through the generic HTTP downloader, so Planet imagery downloads at prep time.
  • Local dev proxy (docker/nginx.conf): TiTiler timeouts + client_max_body_size for chunk uploads.
  • QUICKSTART: DOCKER_GID must be 0 on macOS Docker Desktop.

Azure Maps auth is unchanged (existing anonymous/AAD Client-ID flow).

Spec: spec/features/open-data-catalog/.

Add an Open Data Catalog side panel to the Create Image Layer page so
analysts can browse open disaster imagery (Vantor/Maxar + Planet Open Data),
preview it, and build an image layer without hunting for COG URLs.

Catalog UI (ui/src/Components/OpenDataCatalog/):
- Dynamic discovery + merge of every event across the Vantor S3 STAC and
  Planet Source-Cooperative STAC catalogs; framework-free module with a
  discoverEvents()/fetchEventCatalog() contract (swap-to-backend seam).
- Azure Maps footprints + TiTiler COG preview (works with no Azure Maps key);
  preview tile loading indicator.
- Add a scene to pre/post in one click with source-type + capture-date
  auto-fill; buttons are phase-scoped (pre->Pre, post->Post).
- Two-way map<->list selection: click a footprint to scroll/highlight/expand
  the list row; footprints stay clickable during preview.

Server-side clip AOI:
- Draw a box that sets a layer-level clipBbox ([w,s,e,n] EPSG:4326); imagery
  prep clips the pre/post mosaics via gdalwarp -te/-te_srs (GDAL reprojects
  the bounds). The catalog filters scenes overlapping the AOI and badges
  scenes that fully cover it, so pre/post share one AOI.
- Model ImageLayer.clipBbox + validate_clip_bbox; PutLayer validates/carries
  it; threaded through the imagery-prep config -> ImageryWorkflow ->
  mosaic_imagery. Verified end-to-end (output bounds match the drawn box).

Supporting fixes:
- Allow data.source.coop on the imagery allowlist (UI + hastelib) and route
  non-S3/Blob allowlisted hosts through the generic HTTP downloader, so Planet
  imagery downloads at prep time.
- Local dev proxy (docker/nginx.conf): TiTiler timeouts + client_max_body_size
  for chunk uploads.
- QUICKSTART: DOCKER_GID must be 0 on macOS Docker Desktop.

Azure Maps auth is unchanged (existing anonymous/AAD Client-ID flow).

Spec: spec/features/open-data-catalog/.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@prbatero
prbatero requested review from Copilot and mgmachado July 23, 2026 00:16
@prbatero
prbatero requested a review from a team as a code owner July 23, 2026 00:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an in-app Open Data Catalog explorer on the Create Image Layer flow, letting analysts discover/preview open disaster imagery (Vantor/Maxar + Planet Source Cooperative) and build image layers faster, including an optional server-side clip AOI applied during imagery prep.

Changes:

  • Adds new Open Data Catalog UI (panel + list + Azure Maps footprints + TiTiler preview) and wires it into the Create Image Layer form (including scene-to-pre/post “add” + auto-fill).
  • Introduces ImageLayer.clipBbox end-to-end (UI → PutLayer validation → imagery-prep config → GDAL mosaic clip).
  • Expands imagery URL allowlist to include data.source.coop and routes non-S3/Blob allowlisted URLs through the generic HTTP downloader; updates local dev proxy settings/docs.

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
ui/src/util/validation.js Allowlists data.source.coop imagery hosts in UI URL validation.
ui/src/Components/OpenDataCatalog/SceneListItem.jsx New scene row UI with metadata, AOI badge, and phase-scoped add buttons.
ui/src/Components/OpenDataCatalog/OpenDataCatalogPanel.jsx New catalog side panel: discovery, filtering, list↔map selection sync, clip toolbar.
ui/src/Components/OpenDataCatalog/OpenDataCatalogMap.jsx New Azure Maps footprint rendering + TiTiler preview + rectangle clip drawing.
ui/src/Components/OpenDataCatalog/openDataCatalog.js Framework-free STAC discovery/normalization + TiTiler URL helpers + bbox overlap helpers.
ui/src/Components/CreateEditImageLayerHelper.js Adds clipBbox to form state + helper to add catalog scenes into imagery URL arrays.
ui/src/Components/CreateEditImageLayerForm.jsx Adds “Browse Open Data Catalog” entry point; threads clipBbox into PutLayer payload.
spec/features/open-data-catalog/README.md Feature spec overview and decision log for catalog + server-side clip.
spec/features/open-data-catalog/design.md Detailed design: discovery/preview/clip flow and API contract changes.
spec/features/open-data-catalog/data-model.md Documents clipBbox schema change and allowlist additions.
spec/features/open-data-catalog/user-stories.md User stories and acceptance criteria for catalog and clip workflows.
spec/features/open-data-catalog/test-plan.md Test strategy/coverage matrix for allowlist + clip + manual flows.
spec/features/open-data-catalog/plan.md Implementation plan/status across core/API/UI phases.
spec/features/open-data-catalog/impact-analysis.md Impact/risk analysis across UI/API/workflow/security considerations.
spec/features/open-data-catalog/rollout.md Rollout plan, prerequisites, monitoring, rollback notes.
QUICKSTART.md Documents macOS Docker Desktop DOCKER_GID behavior (set to 0).
hastelib/tests/core/utils/test_url_allowlist.py Adds unit tests for data.source.coop allowlist + validate_clip_bbox.
hastelib/src/hastegeo/workflows/prepare_imagery.py Threads clip_bbox into workflow and mosaic creation.
hastelib/src/hastegeo/core/utils/url_allowlist.py Adds source.coop allowlist and validate_clip_bbox validator.
hastelib/src/hastegeo/core/utils/imagery.py Clips mosaics during GDAL warp when clip_bbox is set.
hastelib/src/hastegeo/core/utils/downloader.py Routes non-S3/Blob allowlisted imagery hosts to generic HTTP downloader.
hastelib/src/hastegeo/core/processors/imagery.py Adds clip_bbox into imagery-prep config payload.
hastelib/src/hastegeo/core/models/projects.py Adds ImageLayer.clipBbox field to model.
docker/nginx.conf Increases body size and TiTiler proxy timeouts for local dev.
docker/docker-compose.yml Adds CLEANUP_CONTAINERS env var to aid debugging spawned containers.
api/hastefuncapi/function_app.py Validates clipBbox in PutLayer (400 on invalid bbox).

Comment thread ui/src/Components/OpenDataCatalog/SceneListItem.jsx
Comment thread ui/src/Components/OpenDataCatalog/SceneListItem.jsx
Comment thread ui/src/util/validation.js
Comment thread hastelib/src/hastegeo/core/utils/url_allowlist.py
- SceneListItem.formatGsd: guard non-finite input so a non-numeric gsd
  renders as absent instead of "NaN m GSD".
- SceneListItem.formatSize: treat only null/undefined/"" as absent so a
  legitimate 0-byte asset formats as "0 B" (was hidden by a falsy check).
- Imagery allowlist host description (ui/util/validation.js and
  hastelib url_allowlist.py): say Source Cooperative "*.source.coop" to
  match what the validator actually allows (endsWith ".source.coop"),
  avoiding misleading error messages.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

RC artifacts ready

All branch deployment references use the same RC tag:

  • hastegeo_version: 1.0.27rc2
  • training_image_tag: 1.0.27rc2
  • imageprep_image_tag: 1.0.27rc2
  • wheel: hastegeo-1.0.27rc2-py3-none-any.whl

The Open Data Catalog fetches the Vantor (S3) and Planet (Source
Cooperative) STAC catalogs + thumbnails directly from the browser. The
deployed Static Web Apps CSP only allowed atlas/openstreetmap/blob hosts,
so connect-src (and img-src) blocked those requests in production
("Refused to connect ... violates the document's Content Security Policy").

Add the two exact hosts to connect-src and img-src in
ui/public/staticwebapp.config.json (and document them in
docs/security-configuration.md):
- https://vantor-opendata.s3.amazonaws.com
- https://data.source.coop

(Longer term, moving discovery behind a same-origin backend proxy would
keep connect-src at 'self'.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants