feat: add scoped collection and document management APIs

[KyleZheng1284]

Description

This PR adds collection and document lifecycle management to the NeMo Retriever service. It provides a logical, scope-isolated API for creating collections, ingesting and replacing documents, tracking asynchronous jobs, querying collection content, and cleaning up resources without exposing physical LanceDB table names or storage locations to clients.

What changed

• Added scope-isolated APIs for managing collections, documents, ingestion jobs, and retrieval.
• Added stable document identities, idempotent ingestion, asynchronous status tracking, and retryable lifecycle cleanup.
• Kept authentication, physical VectorDB identifiers, storage locations, and owned artifacts within the service boundary.
• Added Python client support with API compatibility handling and legacy SSE fallback.
• Added deployment support through Docker Compose and Helm, including non-root artifact storage.
• Added regression tests, API documentation, configuration examples, and operational guidance.

Why

Applications that want to integrate NeMo Retriever (like aiq) need a stable service boundary for managing durable knowledge collections. Previously, consumers had to depend on ingestion-specific behavior or physical VectorDB details.

This change establishes a logical API boundary where:

• clients work with collection names and stable document IDs;
• authentication determines the authorized scope;
• internal table names and storage locations remain service-owned;
• ingestion attempts can be retried without changing the logical document identity;
• collection cleanup and expiration remain recoverable after service restarts.

Validation

Focused regression testing was run from commit edfed55da:

35 passed in 26.06s

The focused coverage includes:

• collection CRUD and pagination;
• scope authorization and cross-scope isolation;
• internal VectorDB authentication;
• idempotent job replay;
• stable document IDs and attempt-ID tracking;
• collection and document cleanup behavior;
• API-version compatibility errors;
• legacy SSE fallback behavior;
• unique OpenAPI operation IDs;
• non-root artifact-directory ownership; and
• collection-management CI and Compose contracts.

The collection workflow was also exercised end to end with collection creation, asynchronous ingestion, job polling, document listing, retrieval, deletion, cleanup, and persistence across service recreation.

Out of scope

The separate TXT/HTML tokenizer fix is intentionally not included in this PR. The integrated validation environment contained that independent patch, but this PR does not modify the tokenizer implementation.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[greptile-apps]

Greptile Summary

This PR introduces scope-isolated collection and document lifecycle management to NeMo Retriever, adding a logical API boundary where clients work with collection names and stable document IDs while physical LanceDB tables and storage locations remain service-owned. The changes address several issues flagged in earlier review rounds — allow_unscoped_dev now defaults to False, the 401/404 scope-oracle is closed, the idempotency-map memory leak is fixed, reconciliation scans outside _write_lock, and the document-ID mismatch for legacy jobs is corrected.

• LanceDBCollectionStore (lancedb_collections.py): 1,211-line new class implementing collection/document CRUD, a reader/writer table-user lease mechanism, retryable lifecycle cleanup, and a background reconciliation loop. The lease mechanism and reconciliation pattern are well-designed, but _collection_write_lock is a single global lock that serializes writes across all collections.
• Gateway router + VectorDB collection endpoints: New proxy routes forward authenticated requests to VectorDB protected by an internal token. Path parameters in the gateway use untyped str where the CollectionName alias should be used.
• JobTracker / JobCreateRequest: Extended with scope, idempotency, document manifest, and IngestOperation fields to support retryable, idempotent multi-file collection ingestion.

Confidence Score: 3/5

The collection management API is well-structured and most previously identified issues have been addressed, but the global _collection_write_lock will serialize all document writes across all collections, limiting throughput of the primary use case at scale.

The global write-serialization bottleneck in LanceDBCollectionStore means only one document can be written at a time across the entire service, directly undermining the multi-collection API. The other findings are lower-stakes and would not block a controlled rollout, but the write-lock issue warrants a fix before production-scale usage.

Files Needing Attention: lancedb_collections.py — the _collection_write_lock design deserves a second pass. routers/collections.py and client.py have minor type-safety gaps worth closing.

Important Files Changed

Filename	Overview
nemo_retriever/src/nemo_retriever/common/vdb/lancedb_collections.py	New 1211-line LanceDB-backed catalog for collection/document lifecycle. Key concern: _collection_write_lock is a global lock that serializes all collection writes regardless of target collection. Reader/writer lease mechanism is sound; reconciliation scan pattern is correct.
nemo_retriever/src/nemo_retriever/service/routers/collections.py	New gateway proxy router forwarding collection/document management requests to VectorDB. Path parameters typed as str instead of the validated CollectionName alias, missing early validation at the gateway boundary.
nemo_retriever/src/nemo_retriever/service/vectordb_app.py	Major refactor: VDB operations moved behind a contract-based VDB interface; collection CRUD endpoints and metrics added. Health endpoint now returns 503 on transient backend exceptions, a behavioral change from the always-200 old implementation.
nemo_retriever/src/nemo_retriever/service/auth.py	Added ScopeAuthorizer with multi-token/scope file support; fixed the previously reported 401/404 oracle. Internal token path added for worker/VectorDB authentication.
nemo_retriever/src/nemo_retriever/service/services/job_tracker.py	Added scope, idempotency map, document manifest, and register_document_idempotent; idempotency-key ownership check in _drop_job_locked addresses the previously reported memory leak.
nemo_retriever/src/nemo_retriever/service/client.py	Added collection/document lifecycle methods and _scope header support. Most async methods now have explicit typed signatures, but aupdate_collection still uses **changes: Any.
nemo_retriever/src/nemo_retriever/common/schemas/collections.py	New shared wire models. CollectionName and DocumentId use a strict pattern addressing the previously reported SQL-injection concern. Models are well-structured.
nemo_retriever/src/nemo_retriever/common/schemas/requests.py	Added manifest, idempotency, and collection fields to JobCreateRequest. _validate_job_contract enforces operation-specific constraints correctly. collection_name now uses the validated CollectionName type.
nemo_retriever/src/nemo_retriever/service/config.py	Added scope/token/internal-VDB-token configuration; allow_unscoped_dev default is now False (fail-closed). Secrets loaded from environment variables and secret files correctly.
nemo_retriever/src/nemo_retriever/service/routers/ingest.py	Extended with collection-aware document handling; _resolve_stable_document_id correctly uses attempt_id for legacy jobs. _prepare_job_work_item consolidates shared upload logic cleanly.

_{Reviews (27): Last reviewed commit: "fix: reconcile agentic queries with VDB ..." | Re-trigger Greptile}

[KyleZheng1284]

@greptileai

[jperez999]

Lets do this in multiple passes

[jperez999]

This looks good to me, aside from the greptile comments that have not been resolved. I had a few questions about the abstract_methods decorator not being used for the new function in adt_vdb. We should probably require those, so everything is on the new setup.