Add Privacy Guard NER engine

docs/documentation/privacy-guard/architecture/entity-processing-engines.md

@@ -294,15 +294,64 @@ Important properties:
 The third-party `regex` backend is used because it can interrupt an individual
 search when the timeout expires.
 
+## NEREngine
+
+`NEREngine` is a resource-backed built-in for general named-entity recognition.
+Its policy configuration owns:
+
+- an ordered non-empty list of printable labels
+- a required finite threshold from zero through one
+- `nested` or `flat` overlap behavior
+- an optional template replacement allowing only literal text and `{entity}`
+
+The deployment registers one immutable `NERResources` bundle containing a
+provider-neutral `NERModel`. The facade has one synchronous
+`predict_entities` operation. Endpoint URLs, model identifiers, chunk
+configuration, device placement, and loaded model objects never enter policy.
+The default built-in registry remains Regex-only; it registers NER only when
+the operator explicitly supplies `ner_resources`.
+
+The package provides two focused facades:
+
+- `NERExtractEndpointModel` sends the exact `POST /v1/extract` JSON contract,
+  bounds response bytes before decoding, performs no retries, and translates
+  expected transport or schema failures without exposing content, endpoint
+  details, or raw collaborator errors.
+- `LocalNERModel` receives an already-loaded object without importing GLiNER,
+  serializes access, processes long text in overlapping code-point windows,
+  rebases offsets, and removes exact chunk-overlap duplicates.
+
+The local facade's lock acquisition is bounded by the shared timeout. Once a
+loaded model call starts, Python's worker-thread architecture cannot preempt
+it; expiration is observed after that call returns.
+
+The engine accepts only returned labels declared by policy, canonicalizes
+case-only differences, rejects invalid spans and scores, retains the
+highest-scoring exact duplicate, and sorts results deterministically. Raw
+scores remain private to duplicate and replacement selection and do not become
+finding confidence or metadata.
+
+Nested detection may return overlapping findings. Replacement keeps every
+finding but selects non-overlapping winners by score, span length, earlier
+start, and configured label order. It projects UTF-8 output size before
+allocating the replacement.
+
+The extract endpoint contract is not the self-hosted Anonymizer GLiNER
+`/v1/chat/completions` contract. Supporting that deployment requires a future
+explicit facade with its own response validation and must not rely on protocol
+autodetection. A future full Anonymizer engine remains separately configured
+because it may own validation, augmentation, or rewriting beyond general NER.
+
 ## Tool-specific custom engines
 
 Tool integrations belong in custom engines until they have a complete,
 production-backed implementation. Privacy Guard does not ship placeholder
 engine types or runtime protocols that merely resemble a third-party tool.
 
-The first NeMo Anonymizer integration will be a custom engine. Its configuration
-and replacement types should preserve Anonymizer's native concepts, while the
-engine itself owns all translation to and from the actual Anonymizer SDK.
+A future full NeMo Anonymizer integration will be a custom engine. Its
+configuration and replacement types should preserve Anonymizer's native
+concepts, while the engine itself owns translation to and from the actual
+Anonymizer SDK.
 
 ## State
 

docs/documentation/privacy-guard/architecture/index.md

@@ -64,8 +64,8 @@ Source paths on these pages are relative to
   policy action. It does not import gRPC or implement an engine's algorithms.
 - `engines/` defines the custom-engine contract, registers engine
   implementations and operator-owned resources, builds the exact Pydantic
-  discriminated union, and contains the built-in Regex implementation. Each
-  engine owns its detection and replacement algorithms.
+  discriminated union, and contains the built-in Regex and resource-backed NER
+  implementations. Each engine owns its detection and replacement algorithms.
 - `config.py` defines ordered stages and the required policy action.
 - top-level `base.py` defines the package-wide strict immutable domain-model
   base.

docs/documentation/privacy-guard/architecture/safety-and-limits.md

@@ -44,6 +44,7 @@ reimplement either.
 | Maximum configured timeout | `MAX_TIMEOUT_SECONDS` | 30 seconds | Server, processor, and `Timeout` |
 | Active processor workers | `MAX_CONCURRENT_PROCESSING` | 4 | Service |
 | Concurrent gRPC calls | `MAX_CONCURRENT_RPCS` | 16 | gRPC server |
+| NER endpoint response | `MAX_NER_ENDPOINT_RESPONSE_BYTES` | 1 MiB | NER endpoint facade |
 
 The gRPC receive allowance is 1 MiB larger than the request body limit for the
 protobuf envelope. The service still enforces the advertised 4 MiB body bound.
@@ -100,6 +101,32 @@ The per-evaluation config `Struct` is limited to 64 KiB. A catalog may satisfy
 Privacy Guard's engine limits yet remain too large for the current upstream
 protocol. Internal caching does not raise that transport ceiling.
 
+## NER configuration and execution limits
+
+| Limit | Constant | Value |
+| --- | --- | ---: |
+| Labels per NER stage | `MAX_NER_LABELS` | 128 |
+| UTF-8 bytes per label | `MAX_NER_LABEL_BYTES` | 256 |
+| Total UTF-8 label bytes | `MAX_NER_LABELS_BYTES` | 16 KiB |
+| Endpoint response bytes | `MAX_NER_ENDPOINT_RESPONSE_BYTES` | 1 MiB |
+
+NER labels must be non-empty printable Unicode without leading or trailing
+whitespace and must be unique after Unicode case folding. Model identity,
+endpoint, and execution chunking are deployment resources rather than policy
+fields.
+
+The endpoint facade bounds response bytes before JSON decoding and performs one
+request without retry or fallback. The local facade processes every supported
+input in overlapping code-point windows and rebases returned offsets. Its
+overlap must be smaller than its chunk length. Operators should choose overlap
+using the longest entity spans expected in the deployment corpus.
+
+Both facades return only normalized label, start, end, and score values. The
+engine rejects unknown labels, invalid or out-of-bounds spans, non-finite or
+out-of-range scores, and excessive result cardinality. Request text, returned
+matched text, raw response bodies, endpoint URLs, credentials, and collaborator
+exception messages are not logged or included in errors.
+
 ## Regex execution safety
 
 `RegexEngine` validates and compiles the complete catalog before accepting

docs/documentation/privacy-guard/architecture/service-boundary.md

@@ -216,9 +216,10 @@ EngineRegistry
   -> gRPC server
 ```
 
-The built-in registry includes `RegexEngine`. Operators register custom engines
-and resource-backed tool integrations before registry finalization, then pass
-that registry to `PrivacyGuardServer`.
+The built-in registry always includes `RegexEngine` and includes `NEREngine`
+only when the operator supplies explicit NER resources. Operators register
+custom engines and other resource-backed tool integrations before registry
+finalization, then pass that registry to `PrivacyGuardServer`.
 
 The registry is an explicit application-scoped dependency, not a global
 singleton. `PrivacyGuardServer` and `PrivacyGuardMiddleware` reject unfinalized

projects/privacy-guard/README.md

@@ -103,6 +103,48 @@ contract.
 
 Privacy Guard owns the catalog schema but maintains no authoritative patterns.
 
+### NEREngine
+
+`NEREngine` is a resource-backed general named-entity recognizer. Policy
+declares an ordered non-empty label list, a required score threshold, nested or
+flat overlap behavior, and an optional constrained replacement template.
+Deployment supplies one provider-neutral model facade through `NERResources`.
+The package includes facades for the explicit GLiNER-compatible
+`POST /v1/extract` contract and for an already-loaded local model; neither
+model identity nor endpoint appears in policy.
+
+The built-in registry remains Regex-only unless NER resources are supplied:
+
+```python
+from privacy_guard.engines import (
+    NERExtractEndpointModel,
+    NERResources,
+)
+from privacy_guard.engines.registry import create_builtin_registry
+
+registry = create_builtin_registry(
+    ner_resources=NERResources(
+        model=NERExtractEndpointModel(
+            endpoint="http://model-host:8002/v1/extract",
+            model="operator-selected-model",
+        )
+    )
+)
+```
+
+NER model scores remain internal and are not mapped to Privacy Guard's
+categorical confidence because model scores are not universally calibrated.
+Case-only returned-label differences are canonicalized to the configured
+label. Exact duplicate spans retain the highest score. Replacement keeps every
+finding while choosing non-overlapping winners by score, span length, start,
+and configured label order.
+
+See the
+[NER engine end-to-end example](https://github.com/NVIDIA/OpenShell-Research/blob/main/projects/privacy-guard/examples/ner-engine/README.md)
+for endpoint and local registry factories. The self-hosted Anonymizer GLiNER
+chat-completions contract is a future dedicated facade rather than an
+auto-detected variant of the extract endpoint.
+
 ## Custom engines
 
 Custom engines are a first-class extension point. Authors declare one typed
@@ -120,8 +162,8 @@ must contain no policy behavior or per-request state and must be safe for
 concurrent use. Resource-free engines omit the second
 `EntityProcessingEngine` generic argument entirely.
 
-The first NeMo Anonymizer integration will be implemented as a custom engine,
-not as a built-in or placeholder abstraction in Privacy Guard.
+A future full NeMo Anonymizer integration remains a separate custom engine,
+not an expansion of the general built-in NER facade.
 
 Application startup registers engines and operator-owned resources, then
 returns one finalized registry:

projects/privacy-guard/examples/ner-engine/README.md

@@ -0,0 +1,98 @@
+# NER engine example
+
+This example registers Privacy Guard's general `ner` engine with either the
+explicit `POST /v1/extract` endpoint contract or an already-downloaded local
+GLiNER model. Policy owns labels, threshold, overlap behavior, replacement,
+and the final action. The registry factory owns the model and its execution
+location.
+
+The initial `nvidia/gliner-PII` checkpoint is optimized for English PII/PHI.
+It is neither a required model nor a built-in entity catalog. Validate the
+chosen model, labels, threshold, and representative inputs before production.
+
+## Endpoint-backed registry
+
+Set the exact endpoint and model identifier. Request text leaves the Privacy
+Guard process, so use only an operator-approved endpoint and protected network.
+The endpoint URL must be an `http` or `https` URL whose path is exactly
+`/v1/extract`; embedded credentials, query strings, and fragments are rejected.
+
+```bash
+cd projects/privacy-guard
+export PYTHONPATH="$PWD/examples/ner-engine"
+export PRIVACY_GUARD_NER_ENDPOINT="http://spark-9107.local:8002/v1/extract"
+export PRIVACY_GUARD_NER_MODEL="nvidia/gliner-PII"
+
+uv run privacy-guard \
+  --registry-factory endpoint_registry:create_registry engines
+uv run privacy-guard \
+  --registry-factory endpoint_registry:create_registry configuration-schema
+uv run privacy-guard \
+  --registry-factory endpoint_registry:create_registry configure-gateway \
+  --host-ip YOUR_HOST_IPV4
+uv run privacy-guard \
+  --registry-factory endpoint_registry:create_registry serve \
+  --listen 0.0.0.0:50051 \
+  --timeout-seconds 30
+```
+
+The endpoint receives `text`, `labels`, `model`, `threshold`, `chunk_length`,
+`overlap`, and `flat_ner`. Privacy Guard performs no retry, fallback, health
+check, or protocol detection.
+
+## Direct local registry
+
+The direct example deliberately keeps GLiNER and PyTorch out of Privacy Guard's
+base dependencies. Install the characterized library version in the deployment
+environment and download the model through an operator-controlled workflow:
+
+```bash
+uv pip install "gliner==0.2.27"
+export PYTHONPATH="$PWD/examples/ner-engine"
+export PRIVACY_GUARD_NER_MODEL_PATH="/absolute/path/to/downloaded/model"
+
+uv run privacy-guard \
+  --registry-factory local_registry:create_registry engines
+uv run privacy-guard \
+  --registry-factory local_registry:create_registry serve \
+  --listen 0.0.0.0:50051 \
+  --timeout-seconds 30
+```
+
+`local_files_only=True` prevents an implicit model download. Local calls are
+serialized because loaded-model thread safety and GPU memory behavior are not
+assumed. Long input is processed in overlapping Unicode code-point windows,
+with global offset rebasing and exact duplicate removal. A running model call
+cannot be preempted by Privacy Guard's worker thread; timeout is checked after
+the call returns.
+
+`PRIVACY_GUARD_NER_CHUNK_LENGTH` and
+`PRIVACY_GUARD_NER_CHUNK_OVERLAP` optionally override the defaults of 384 and
+128. The endpoint interprets these according to its contract. The local facade
+uses them as code-point window sizes and requires overlap to be smaller than
+chunk length.
+
+## Exercise policy actions
+
+Use [`policy.yaml`](policy.yaml) as the OpenShell policy source. The extracted
+[`privacy-guard-config.yaml`](privacy-guard-config.yaml) is convenient for
+direct middleware tests. Its configured replacement may stay present for every
+action:
+
+- `detect` allows the original request and reports findings.
+- `block` denies a request when a configured entity is found.
+- `replace` sends the body after deterministic, non-overlapping template
+  replacement.
+
+Change only `on_detection.action` to exercise these modes. Nested detection
+returns all valid model spans. Replacement keeps all findings but selects
+non-overlapping winners by score, span length, start offset, and configured
+label order.
+
+Scores are not exposed as categorical confidence because model scores are not
+universally calibrated. Tune the required `threshold` using use-case-specific
+evaluation.
+
+The Anonymizer self-hosted GLiNER `/v1/chat/completions` contract is not
+accepted by this example. It remains a dedicated future adapter so Privacy
+Guard never guesses a provider protocol from a URL.

projects/privacy-guard/examples/ner-engine/endpoint_registry.py

@@ -0,0 +1,49 @@
+"""Registry factory for the explicit ``POST /v1/extract`` NER endpoint."""
+
+from __future__ import annotations
+
+import os
+
+from privacy_guard.engines import NERExtractEndpointModel, NERResources
+from privacy_guard.engines.registry import EngineRegistry
+
+
+def create_registry() -> EngineRegistry:
+    """Create a built-in registry backed by the configured NER endpoint."""
+    endpoint = _required_environment("PRIVACY_GUARD_NER_ENDPOINT")
+    model = _required_environment("PRIVACY_GUARD_NER_MODEL")
+    resources = NERResources(
+        model=NERExtractEndpointModel(
+            endpoint=endpoint,
+            model=model,
+            chunk_length=_environment_integer(
+                "PRIVACY_GUARD_NER_CHUNK_LENGTH",
+                default=384,
+            ),
+            overlap=_environment_integer(
+                "PRIVACY_GUARD_NER_CHUNK_OVERLAP",
+                default=128,
+            ),
+        )
+    )
+    return EngineRegistry(
+        include_builtin_engines=True,
+        ner_resources=resources,
+    ).finalize()
+
+
+def _required_environment(name: str) -> str:
+    value = os.environ.get(name)
+    if value is None or not value:
+        raise RuntimeError(f"Required environment variable {name} is not set")
+    return value
+
+
+def _environment_integer(name: str, *, default: int) -> int:
+    value = os.environ.get(name)
+    if value is None:
+        return default
+    try:
+        return int(value)
+    except ValueError:
+        raise RuntimeError(f"Environment variable {name} must be an integer") from None

projects/privacy-guard/examples/ner-engine/local_registry.py

@@ -0,0 +1,50 @@
+"""Registry factory for an already-downloaded local GLiNER model."""
+
+from __future__ import annotations
+
+import os
+from importlib import import_module
+
+from privacy_guard.engines import LocalNERModel, NERResources
+from privacy_guard.engines.registry import EngineRegistry
+
+
+def create_registry() -> EngineRegistry:
+    """Load one local model at startup and create the built-in NER registry."""
+    model_path = _required_environment("PRIVACY_GUARD_NER_MODEL_PATH")
+    gliner_type = getattr(import_module("gliner"), "GLiNER")
+    loaded_model = gliner_type.from_pretrained(model_path, local_files_only=True)
+    resources = NERResources(
+        model=LocalNERModel(
+            model=loaded_model,
+            chunk_length=_environment_integer(
+                "PRIVACY_GUARD_NER_CHUNK_LENGTH",
+                default=384,
+            ),
+            overlap=_environment_integer(
+                "PRIVACY_GUARD_NER_CHUNK_OVERLAP",
+                default=128,
+            ),
+        )
+    )
+    return EngineRegistry(
+        include_builtin_engines=True,
+        ner_resources=resources,
+    ).finalize()
+
+
+def _required_environment(name: str) -> str:
+    value = os.environ.get(name)
+    if value is None or not value:
+        raise RuntimeError(f"Required environment variable {name} is not set")
+    return value
+
+
+def _environment_integer(name: str, *, default: int) -> int:
+    value = os.environ.get(name)
+    if value is None:
+        return default
+    try:
+        return int(value)
+    except ValueError:
+        raise RuntimeError(f"Environment variable {name} must be an integer") from None