feat(usfm): move async export to Azure Blob with SAS downloads and reliable jobs

[henrique221]

Closes #196.

Summary

Implements the async-export hardening per Kasey's decision (Azure Blob Storage + signed URLs). The async pipeline is still not wired to the UI — this makes it safe to enable. All four audit defects are addressed:

Defect	Fix
Failed jobs marked completed (retries never fired)	Worker runs batchSize: 1 and rethrows; pg-boss retries apply; terminal failures land on a new usfm-export-dlq dead-letter queue
Worker-local disk invisible to the API (downloads 404)	Worker streams the ZIP into Azure Blob (Azurite in local compose); GET /downloads/{filename} 302-redirects to a 15-min SAS URL after an ownership check
Whole ZIP buffered in memory (OOM risk)	Streaming upload (4 MB blocks) with a byte-counting transform — memory is flat regardless of export size
No idempotency / owner binding; hardcoded requestedBy	requestedBy = authenticated user id; /jobs/{id} + /downloads/{filename} 404 for anyone but the requester (blob metadata carries the owner); singletonKey + exclusive queue policy collapse duplicates into 409; expireInSeconds 3600 → 600

Design notes

• ensureExportQueues() (in lib/queue.ts) self-converges queue config on boot: createQueue no-ops on existing queues and policy is immutable, so a queue created by older code is dropped and recreated with the exclusive policy (safe pre-enablement), and tunable options converge via updateQueue.
• Not configured ⇒ fail loud: without AZURE_STORAGE_CONNECTION_STRING, the async endpoints respond 503 and the worker refuses to start. The sync export is unaffected.
• Local dev: compose gains an azurite service (--skipApiVersionCheck for SDK/emulator version skew); .env.example documents the well-known dev connection string. file-storage.ts / EXPORTS_DIR are removed.
• 4 new worker unit tests (failure propagation — the core bug — plus upload metadata/cleanup); 105/105 suite passes.

E2E (compose + Azurite, full pipeline)

• 202 enqueue → immediate duplicate → 409
• job completed → GET /downloads/… → 302 with SAS Location → fetch → valid ZIP (PK, 552 bytes)
• translator (non-owner) → 404 on both /jobs/{id} and /downloads/…; owner still 302
• Azurite stopped → job active → retry (failure logged for App Insights) → Azurite restarted → retry recovered to completed

⚠️ Deploy prerequisite (infra)

Hosted environments need a storage account + container and AZURE_STORAGE_CONNECTION_STRING (must include an AccountKey, which signs the SAS URLs) set for both the API and the worker, plus optional EXPORTS_CONTAINER (default usfm-exports). Until then the async endpoints 503 — nothing else changes.

Test plan

• npm run typecheck / npm run lint / npm run format:check
• npm run test — 105/105
• Full E2E against compose + Azurite (above)

Screenshots

1) OpenAPI — async endpoint contract (202 / 409 / 503)

The Scalar reference for POST /project-units/{projectUnitId}/usfm/async documents the new 409 Conflict (duplicate export collapsed via singletonKey) and 503 Service Unavailable (storage not configured) alongside the 202.

---

2) OpenAPI — download is now a 302 redirect to a signed URL

GET /downloads/{filename} documents 302 (redirect to a short-lived SAS URL) and 503, replacing the old direct-zip 200.

---

3) Live pipeline — enqueue → 409 dedupe → completed → 302 SAS → valid ZIP

Full pipeline against the running compose stack (API + worker + Azurite): 202 with jobId; an immediate identical request gets 409; the job completes (552 bytes, 74 ms); the download responds 302 Found with an Azurite SAS Location; fetching it yields a valid ZIP (PK magic).

---

4) Failure & retry recovery — the core fix

Azurite stopped before enqueue: the job goes created → active → retry (upload fails, error logged). After restarting Azurite, the 60s retry fires and the job completes at t+70s with a downloadable ZIP. Under the old code this exact scenario was marked completed with no file and no retry.

---

Summary by CodeRabbit

New Features

• Async USFM exports are stored in Cloudflare R2 and delivered through short-lived signed download links.
• Export requests support deduplication, expiration, streaming uploads, automatic cleanup, and resilient retries.

Bug Fixes

• Unavailable storage returns 503, while duplicate requests return 409.
• Export access is restricted to the requester, with improved retry and dead-letter handling.

Documentation

• Updated R2 configuration, EU storage, endpoint, local development, and cleanup guidance.

Chores

• Removed local export-directory storage and related container mounts.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 8eadb7bc-e07b-41e4-81a2-def15fe478f7

📥 Commits

Reviewing files that changed from the base of the PR and between ed54102 and f1c0a18.

📒 Files selected for processing (5)

• docs/proposals/security-and-export-hardening/03-async-export-pipeline-hardening.md
• docs/proposals/security-and-export-hardening/README.md
• src/index.ts
• src/lib/blob-storage.test.ts
• src/lib/blob-storage.ts

🚧 Files skipped from review as they are similar to previous changes (3)

• src/index.ts
• docs/proposals/security-and-export-hardening/README.md
• docs/proposals/security-and-export-hardening/03-async-export-pipeline-hardening.md

---

📝 Walkthrough

Walkthrough

This PR replaces local async USFM export files with Cloudflare R2 storage. It adds R2 configuration and cleanup, updates queue and worker retry behavior, and changes export routes to enforce ownership and signed download redirects.

Changes

R2-backed async USFM export

Layer / File(s)	Summary
R2 configuration and runtime wiring .env.example, src/env.ts, compose.yaml, package.json, src/index.ts, src/workers/standalone-worker.ts	Adds R2 settings and AWS S3 dependencies, removes local export mounts, verifies storage at startup, and schedules R2 cleanup.
R2 export storage operations src/lib/blob-storage.ts, src/lib/blob-storage.test.ts	Adds bucket validation, streaming uploads, metadata lookup, signed URLs, and expired-object cleanup with tests.
Queue reconciliation and retry policy src/lib/queue.ts	Adds the export DLQ, exclusive queue reconciliation, retry settings, expiry, and requester typing.
Streaming worker execution src/workers/usfm-export.worker.ts, src/workers/usfm-export.worker.test.ts	Streams ZIP output to R2, processes one job per invocation, rethrows failures for retries, and tests success and failure paths.
Export ownership and download redirects src/domains/usfm/usfm.route.ts	Adds storage availability checks, duplicate request handling, requester-bound access, and signed download redirects.
Pipeline documentation updates docs/proposals/security-and-export-hardening/*	Updates implementation status and documents the R2-based export pipeline.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

• eten-tech-foundation/fluent-api#224: Updates shared Cloudflare R2/S3 configuration for a separate audio storage domain.

Suggested reviewers: anumonachan, vipinpaul

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant USFMRoute
  participant PgBoss
  participant USFMExportWorker
  participant BlobStorage
  Client->>USFMRoute: POST async export request
  USFMRoute->>PgBoss: enqueue requester-bound job
  PgBoss->>USFMExportWorker: deliver one export job
  USFMExportWorker->>BlobStorage: upload ZIP stream
  BlobStorage-->>USFMExportWorker: filename, sizeBytes, expiresAt
  Client->>USFMRoute: GET download endpoint
  USFMRoute->>BlobStorage: validate metadata and generate signed URL
  BlobStorage-->>USFMRoute: signed download URL
  USFMRoute-->>Client: 302 redirect

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title describes the async export storage migration and reliable job handling, which match the main objectives, although the implementation uses R2 instead of Azure Blob.
Linked Issues check	✅ Passed	The implementation addresses #196 by enabling retries, shared R2 storage, presigned downloads, streaming uploads, idempotency, ownership binding, and shorter job expiry.
Out of Scope Changes check	✅ Passed	The changes support #196 through storage, queue, worker, route, configuration, dependency, test, and documentation updates; no unrelated changes are evident.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/196-async-export-hardening

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 7

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@compose.yaml`:
- Around line 19-47: The Azurite service image is unpinned and currently pulls
the implicit latest tag, so update the azurite image reference in compose.yaml
to a specific version to match the repo’s reproducibility convention. Keep the
existing azurite service configuration intact, but change only the image
declaration for azurite so the dev environment consistently uses the same
emulator version across rebuilds.

In `@docs/proposals/security-and-export-hardening/README.md`:
- Line 23: The README intro is out of sync with the Ticket 3 status shown in the
table. Update the opening status block in the proposal index so it matches the
current wording for Ticket 3 being implemented, and keep the intro aligned with
the ticket summary in the table. Use the existing proposal index text near the
status block and the Ticket 3 entry to make the wording consistent.

In `@src/domains/usfm/usfm.route.ts`:
- Around line 388-399: The usfm export deduplication key in usfm.route.ts is too
broad because singletonKey only uses projectUnitId and bookIds, so requests from
different users can collide. Update the singletonKey logic in the export flow to
also include the requester identity from requestedBy/user metadata, so boss.send
only collapses duplicate exports for the same user. Keep the existing dedupe
behavior for identical requests from one requester, but ensure different users
get distinct jobs and jobId values even when the unit and book selection match.

In `@src/lib/blob-storage.ts`:
- Around line 150-169: deleteExpiredExports currently performs a full
client.listBlobsFlat() sweep and sequential deleteBlob calls on every cleanup
cycle, which does not scale as export volume grows. Refactor the blob cleanup
path in deleteExpiredExports (and its callers in the cleanup scheduling code) to
avoid application-level polling, and plan to move TTL enforcement to an Azure
Blob lifecycle management policy so expiry is handled server-side instead of by
hourly scans.

In `@src/lib/queue.ts`:
- Around line 68-91: `ensureExportQueues` currently deletes
`QUEUE_NAMES.USFM_EXPORT` whenever `getQueue` shows a non-exclusive policy, but
`deleteQueue` can fail once jobs already exist. Update the recreation logic in
`ensureExportQueues` to guard the `deleteQueue` call with an explicit
empty-queue check or a migration/drain step before recreating `usfm-export`, and
keep the existing `createQueue`/`updateQueue` flow intact.

In `@src/workers/standalone-worker.ts`:
- Around line 68-72: The hourly cleanup is duplicated in both the worker and the
API server, which causes redundant shared-blob deletions and spurious cleanup
errors when both race on deleteExpiredExports(). Update the ownership so only
one process schedules the setInterval cleanup (for example, keep it in
standalone-worker.ts or move it to src/index.ts, but not both), and ensure the
other startup path does not register the interval. Use the deleteExpiredExports
and cleanupInterval symbols to locate and remove the duplicate scheduling.
- Around line 60-64: The startup reconciliation in ensureExportQueues(boss) is
destructive because it deletes usfm-export before recreating it whenever the
policy is not exclusive, which can break existing jobs and race across the two
boot paths. Update ensureExportQueues and its callers in src/index.ts and
src/workers/standalone-worker.ts so only one owner performs the migration, or
make the delete/recreate flow idempotent and retry-safe. Preserve any existing
queue state if possible, and use the existing initializeQueue,
ensureExportQueues, and registerUSFMExportWorker flow to keep startup
non-destructive.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: cb9c3c9f-4df9-4039-bd8a-6efe7e2f1566

📥 Commits

Reviewing files that changed from the base of the PR and between cb105b7 and b0f04b0.

⛔ Files ignored due to path filters (1)

• package-lock.json is excluded by !**/package-lock.json

📒 Files selected for processing (14)

• .env.example
• compose.yaml
• docs/proposals/security-and-export-hardening/03-async-export-pipeline-hardening.md
• docs/proposals/security-and-export-hardening/README.md
• package.json
• src/domains/usfm/usfm.route.ts
• src/env.ts
• src/index.ts
• src/lib/blob-storage.ts
• src/lib/file-storage.ts
• src/lib/queue.ts
• src/workers/standalone-worker.ts
• src/workers/usfm-export.worker.test.ts
• src/workers/usfm-export.worker.ts

💤 Files with no reviewable changes (1)

• src/lib/file-storage.ts

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/queue.ts`:
- Around line 85-91: The pending-job guard in queue policy recreation is using
the wrong pg-boss stats fields, so it never detects queued work. Update the
logic in the queue stats handling around getQueueStats and pendingJobs to use
the actual returned fields from pg-boss, namely queuedCount, activeCount, and
deferredCount, instead of created, retry, and active. Keep the warning path in
the same recreate guard so the delete+recreate flow is skipped whenever any jobs
are still pending.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 47508c81-924f-4aee-bbae-ce3951ab88b1

📥 Commits

Reviewing files that changed from the base of the PR and between b0f04b0 and 45ea0d5.

📒 Files selected for processing (6)

• compose.yaml
• docs/proposals/security-and-export-hardening/README.md
• src/domains/usfm/usfm.route.ts
• src/index.ts
• src/lib/blob-storage.ts
• src/lib/queue.ts

[kaseywright]

The decision has been made to use CloudFlare R2 as cloud storage for the project. We should ensure that the GDPR jurisdiction is set so that the files at rest in the R2 buckets are located in known regions in Europe to start with. Some of these settings might be out of scope for this PR while integrating CloudFlare.

[henrique221]

swapped azure blob → cloudflare r2 for the async export path. it's on the s3-compatible aws sdk now — streaming multipart upload, headobject metadata, same app-level ttl sweep, presigned downloads still 15 min. creds are optional so the endpoints still 503 when r2 isn't set, and the sync export path is untouched.

for gdpr: the client is pinned to the eu-jurisdiction endpoint via env (R2_JURISDICTION defaults to eu → {account}.eu.r2.cloudflarestorage.com) so data at rest stays in the eu. creating the bucket in that jurisdiction is the one thing code can't do — it's set at bucket-creation time in the dashboard/IaC — so i documented it in .env.example as the infra step (that's the "some settings out of scope" bit you called out). credential var names line up with the audio-record r2 integration. give it another look when you get a sec?

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@docs/proposals/security-and-export-hardening/03-async-export-pipeline-hardening.md`:
- Around line 9-12: Align the proposal’s 503 documentation with the implemented
isBlobStorageConfigured() readiness check by either limiting the statement to
missing R2 credentials or extending bucket validation to require a provisioned
R2_EXPORTS_BUCKET. If adding validation, update the related readiness tests so
absent or invalid bucket configuration returns 503 before storage operations.

In `@src/lib/blob-storage.ts`:
- Around line 56-79: Update src/lib/blob-storage.ts lines 56-79 in getS3Client
and buildR2Endpoint to use the configured S3-compatible endpoint when present,
requiring R2_ACCOUNT_ID only for the Cloudflare R2 fallback; update src/env.ts
lines 91-101 to declare the optional, URL-validated endpoint variable, add a
MinIO URL example in .env.example lines 46-75, and update compose.yaml lines
37-39 and 73-75 with the endpoint setting and local-development guidance
requiring it for MinIO.
- Around line 187-190: Update generateExportDownloadUrl to derive the export’s
remaining lifetime from its creation timestamp, cap the signed URL duration at
SAS_TTL_MINUTES while never exceeding the one-hour export TTL, and reject
missing or already elapsed timestamps. Reuse getExportBlobInfo or the existing
export metadata source to obtain createdOn, and pass the resulting positive
remaining duration to getSignedUrl.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 5e6ddcf7-efa3-41af-9203-4d8422ca5f41

📥 Commits

Reviewing files that changed from the base of the PR and between 45ea0d5 and 3fc1022.

⛔ Files ignored due to path filters (1)

• package-lock.json is excluded by !**/package-lock.json

📒 Files selected for processing (11)

• .env.example
• compose.yaml
• docs/proposals/security-and-export-hardening/03-async-export-pipeline-hardening.md
• package.json
• src/domains/usfm/usfm.route.ts
• src/env.ts
• src/index.ts
• src/lib/blob-storage.test.ts
• src/lib/blob-storage.ts
• src/lib/queue.ts
• src/workers/standalone-worker.ts

🚧 Files skipped from review as they are similar to previous changes (4)

• src/index.ts
• src/workers/standalone-worker.ts
• src/lib/queue.ts
• src/domains/usfm/usfm.route.ts

[kaseywright]

Review: async export → R2 hardening

Verified locally on this branch: typecheck, lint, and test (183/183) all pass. Overall the four audit defects from #196 are genuinely fixed — streaming upload, batchSize:1 + rethrow + DLQ, owner-bound jobs/downloads, singletonKey dedupe. Requesting changes on one blocking item; the other two need action before/around merge as noted.

Blocking

1. Startup hard-fails the whole API if R2 is misconfigured — for a feature that isn't wired to the UI yet (src/index.ts:15-21)

if (isBlobStorageConfigured()) {
  await initializeBlobStorage();
} else {
  logger.warn(...);
}

initializeBlobStorage() throws on HeadBucketCommand failure, and that throw propagates to startServer()'s outer catch, which calls process.exit(1). Once ops sets the three R2_* vars, a bucket typo, a rotated key, or a transient R2 outage takes down the entire API — not just async export. Since this pipeline is explicitly not exposed to fluent-web yet, that's a disproportionate blast radius for a pre-enablement feature.

Please degrade this to a warning (matching the "credentials unset" path) instead of blocking boot, so a storage misconfiguration only 503s the async export endpoints rather than the whole service.

Needs discussion (not blocking this PR, but let's align before it lands elsewhere)

2. usfm-export-dlq has no consumer (src/lib/queue.ts:75,86)

The dead-letter queue is created and wired as the deadLetter target, but nothing reads from it — no worker, no alert, no count/log distinguishing "exhausted retries, moved to DLQ" from an ordinary retry attempt. Terminal failures are still visible via the per-attempt logger.error in usfm-export.worker.ts, so this isn't silent, but there's no positive signal when a job actually lands in the DLQ. Let's discuss a general strategy for dead-letter queues across the worker fleet (this is likely to recur beyond USFM export) rather than solving it one-off here.

Required before merge

3. Stale doc: README table still says "Azure Blob" (docs/proposals/security-and-export-hardening/README.md:24)

The linked 03-async-export-pipeline-hardening.md was correctly updated to describe the R2 migration, but the summary table row still reads "Implemented — Azure Blob + SAS, worker retries, owner binding (#196)". Please update that line to reflect Cloudflare R2 before merge so the two docs don't contradict each other.

[henrique221]

addressed the review:

1 (blocking) — boot blast radius: fixed in e3c15b4. the api entrypoint now calls verifyBlobStorageOnBoot(), which never throws — creds unset keeps the existing warn + 503 path, and a failed bucket check logs a warning and lets boot continue, with failures surfacing per-job in the worker instead. the dedicated export worker keeps fail-fast initializeBlobStorage, since storage is load-bearing there. covered by 3 new unit tests (creds unset / bucket reachable / bucket unreachable → resolves false, no throw).

3 (stale doc): fixed in f1c0a18 — the readme table row now reads cloudflare r2 (eu) + presigned downloads. the blocking fix also invalidated the "process refuses to boot" wording in the 03 doc, so that sentence was re-worded in e3c15b4.

2 (dlq consumer): agreed this should be a fleet-wide decision rather than a one-off here — the ai-suggestion trigger queue will hit the exact same question. happy to open a follow-up issue to track the dlq strategy discussion (dedicated consumer vs. alerting on dlq depth vs. periodic sweep-and-log) so it does not get lost.

branch is also updated with main (12585ac). suite after the changes: typecheck + lint clean, 186/186 tests.

[henrique221]

opened #256 to track the fleet-wide dlq strategy discussion (consumer vs. depth alerting vs. sweep-and-log, plus retention) — kept out of this pr per the review.

[kaseywright]

Verified the two fixup commits (e3c15b4, f1c0a18) against the blocking review:

• #1 (boot failure on R2 misconfig): fixed correctly via verifyBlobStorageOnBoot(), which never throws — the API stays up and only async export degrades on an unreachable bucket. The dedicated worker correctly keeps its fail-fast behavior since storage is load-bearing there. New unit tests cover all three paths (unconfigured, reachable, unreachable).
• #3 (stale doc): README table now correctly says Cloudflare R2.

typecheck, lint, and test (186/186) all pass locally on the updated branch.

#2 (dead-letter queue strategy) remains open as a separate, non-blocking discussion per my earlier comment — not something to solve in this PR. Approving.