diff --git a/README.md b/README.md index bbbaa22..556184c 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,14 @@ router. `civiccore.search` now ships normalization and fusion helpers, but not a full search engine or indexer. `civiccore.notifications` now ships notice deadline and compliance helpers, but not delivery queues or outbound notification orchestration. +`civiccore.platform` now ships the Windows-local desktop contracts for module +manifests, install-profile validation, operator health summaries, durable local +task envelopes, backup/restore manifests, and installer/runtime action results. +It also ships the PostgreSQL-backed `civiccore_local_tasks` migration, async +queue helpers, and `python -m civiccore.tasks.worker` entry point for the local +desktop worker. Product modules still own task handlers and UI, while CivicCore +owns shared validation, queue state, retry semantics, and plain-English +operator state. `civiccore.verification` now ships the first release-evidence helper surface, while sovereignty verification remains future work. `civiccore.connectors` now also ships shared local-payload import @@ -83,7 +91,10 @@ on top of shared vendor delta request planning plus reusable no-network mock-city proof contracts for agenda vendors, municipal OIDC, and backup retention/off-host storage, on top of shared live connector sync retry/circuit-breaker primitives, including run-result normalization, operator health copy, retry delay policy, and async HTTP retry, -on top of shared persisted audit-log hash and verification helpers for +on top of shared Windows-local module/runtime contracts for the desktop shell, +including no-Docker/no-WSL manifest validation, plain-English health summaries, +PostgreSQL-backed local task queue helpers, and backup/restore checksum +manifests, on top of shared persisted audit-log hash and verification helpers for database-backed module audit rows on top of shared trusted-header auth config loading and proxy-source enforcement helpers on top of shipped trusted-header auth helpers on top of shipped @@ -345,7 +356,35 @@ assert verify_persisted_audit_chain([ These APIs are deliberately offline-first. They do not provide JWT issuance, SSO, user directories, credential storage, vendor-specific network -adapters, worker scheduling, legal determinations, or vendor write-back. +adapters, worker execution, legal determinations, or vendor write-back. + +## Windows-local platform contracts + +`civiccore.platform` exposes the shared contracts and queue helpers the +CivicSuite Windows desktop shell uses to keep future modules pluggable without +making clerks learn infrastructure: + +```python +from civiccore.platform import ( + ModuleManifest, + build_module_registry, + PlatformHealthCheck, + summarize_platform_health, + LocalTaskEnvelope, + enqueue_local_task, + claim_next_local_task, + record_task_attempt, + run_one_local_task, + build_backup_manifest, + plan_restore, +) +``` + +For the `windows_local` install profile, module manifests cannot require +Docker, WSL, Linux shell setup, or a terminal-only operator path. CivicCore also +ships task envelopes, a durable PostgreSQL task table, async queue helpers, and +a worker CLI so downstream modules share retry, health, checksum, and +restore-safety semantics while registering their own task handlers. ## Document ingestion @@ -534,9 +573,10 @@ router integration, and persistence orchestration remain future work. ## Scheduling helper `civiccore.scheduling` exposes the shared cron expression contract used by -module background jobs. Modules keep their own Celery/worker/runtime wiring, -but should reuse this validation so one-minute accidental or adversarial -schedules are blocked consistently across CivicSuite. +module background jobs. `civiccore.platform` exposes the local task envelope, +PostgreSQL queue helpers, and retry contract. Modules keep their own task +handlers, but should reuse these helpers so one-minute accidental or adversarial +schedules and task retry behavior are handled consistently across CivicSuite. ```python from civiccore.scheduling import compute_next_sync_at, validate_cron_expression diff --git a/civiccore/__init__.py b/civiccore/__init__.py index c5e1583..1d85980 100644 --- a/civiccore/__init__.py +++ b/civiccore/__init__.py @@ -112,6 +112,53 @@ next_profile_prompt, parse_profile_answer, ) +from civiccore.platform import ( + WINDOWS_LOCAL_BLOCKED_RUNTIME_KINDS, + BackupItem, + BackupManifest, + BackupRestoreAction, + BackupRestorePlan, + BackupValidationResult, + LocalRuntimeProfile, + LocalTask, + LocalTaskEnvelope, + LocalTaskResult, + ModuleBackupHook, + ModuleDependency, + ModuleHealthCheck, + ModuleManifest, + ModuleMigration, + ModuleModelRequirement, + ModulePermission, + ModuleRegistryEntry, + ModuleRegistryState, + ModuleRoute, + ModuleRuntimeRequirement, + ModuleService, + PlatformHealthCheck, + PlatformHealthStatus, + PlatformHealthSummary, + RuntimeActionResult, + TaskQueueSummary, + TaskHandler, + TaskRetryPolicy, + build_backup_manifest, + build_module_registry, + can_run_task, + claim_next_local_task, + complete_local_task, + enqueue_local_task, + fail_local_task, + next_retry_at, + plan_restore, + record_task_attempt, + run_one_local_task, + summarize_platform_health, + summarize_task_queue, + task_row_to_envelope, + validate_backup_manifest, + validate_windows_local_manifest, +) from civiccore.provenance import ( CitationTarget, DocumentMetadata, @@ -245,6 +292,40 @@ "ModuleEnablement", "OnboardingField", "OnboardingProgress", + "BackupItem", + "BackupManifest", + "BackupRestoreAction", + "BackupRestorePlan", + "BackupValidationResult", + "LocalRuntimeProfile", + "LocalTask", + "LocalTaskEnvelope", + "LocalTaskResult", + "ModuleBackupHook", + "ModuleDependency", + "ModuleHealthCheck", + "ModuleManifest", + "ModuleMigration", + "ModuleModelRequirement", + "ModulePermission", + "ModuleRegistryEntry", + "ModuleRegistryState", + "ModuleRoute", + "ModuleRuntimeRequirement", + "ModuleService", + "PlatformHealthCheck", + "PlatformHealthStatus", + "PlatformHealthSummary", + "RuntimeActionResult", + "TaskQueueSummary", + "TaskHandler", + "TaskRetryPolicy", + "WINDOWS_LOCAL_BLOCKED_RUNTIME_KINDS", + "build_backup_manifest", + "build_module_registry", + "can_run_task", + "claim_next_local_task", + "complete_local_task", "access_level_allows", "completed_profile_fields", "compute_onboarding_status", @@ -269,7 +350,13 @@ "normalize_search_query", "normalize_search_text", "min_interval_minutes", + "enqueue_local_task", + "fail_local_task", + "next_retry_at", "parse_profile_answer", + "plan_restore", + "record_task_attempt", + "run_one_local_task", "roles_grant_access", "search_text_matches_query", "reciprocal_rank_fusion", @@ -282,8 +369,13 @@ "ingest_structured_record", "register_handler", "validate_cited_sentences", + "summarize_platform_health", + "summarize_task_queue", + "task_row_to_envelope", + "validate_backup_manifest", "validate_cron_expression", "validate_fernet_key_setting", + "validate_windows_local_manifest", "validate_odbc_connection_string", "validate_password_setting", "validate_secret_setting", diff --git a/civiccore/migrations/versions/civiccore_0003_local_task_queue.py b/civiccore/migrations/versions/civiccore_0003_local_task_queue.py new file mode 100644 index 0000000..fdec2b6 --- /dev/null +++ b/civiccore/migrations/versions/civiccore_0003_local_task_queue.py @@ -0,0 +1,44 @@ +"""CivicCore migration 0003 - local task queue.""" + +from __future__ import annotations + +import sqlalchemy as sa +from sqlalchemy.dialects import postgresql + +from civiccore.migrations.guards import idempotent_create_index, idempotent_create_table + + +revision = "civiccore_0003_local_task_queue" +down_revision = "civiccore_0002_llm" +branch_labels = None +depends_on = None + + +def upgrade() -> None: + idempotent_create_table( + "civiccore_local_tasks", + sa.Column("task_id", sa.String(128), primary_key=True), + sa.Column("module_id", sa.String(64), nullable=False), + sa.Column("task_type", sa.String(100), nullable=False), + sa.Column("status", sa.String(20), nullable=False, server_default="queued"), + sa.Column( + "payload", + postgresql.JSONB(astext_type=sa.Text()), + nullable=False, + server_default=sa.text("'{}'::jsonb"), + ), + sa.Column("attempt_count", sa.Integer(), nullable=False, server_default="0"), + sa.Column("queued_at", sa.DateTime(timezone=True), nullable=False, server_default=sa.text("now()")), + sa.Column("available_at", sa.DateTime(timezone=True), nullable=True), + sa.Column("last_error", sa.Text(), nullable=True), + sa.Column("idempotency_key", sa.String(255), nullable=True, unique=True), + sa.Column("audit_subject_id", sa.String(255), nullable=True), + ) + idempotent_create_index("ix_civiccore_local_tasks_module", "civiccore_local_tasks", ["module_id"]) + idempotent_create_index("ix_civiccore_local_tasks_status", "civiccore_local_tasks", ["status"]) + idempotent_create_index("ix_civiccore_local_tasks_type", "civiccore_local_tasks", ["task_type"]) + + +def downgrade() -> None: + """No-op; local task queue data is preserved for point-in-time restore.""" + return None diff --git a/civiccore/models/__init__.py b/civiccore/models/__init__.py index 450d112..f960306 100644 --- a/civiccore/models/__init__.py +++ b/civiccore/models/__init__.py @@ -1,11 +1,13 @@ """CivicCore shared SQLAlchemy ORM model exports.""" from civiccore.ingest.models import DataSource, Document, DocumentChunk, IngestionStatus, SourceType +from civiccore.platform.task_queue import LocalTask __all__ = [ "DataSource", "Document", "DocumentChunk", "IngestionStatus", + "LocalTask", "SourceType", ] diff --git a/civiccore/platform/__init__.py b/civiccore/platform/__init__.py new file mode 100644 index 0000000..e0db969 --- /dev/null +++ b/civiccore/platform/__init__.py @@ -0,0 +1,106 @@ +"""Windows-local platform contracts for CivicSuite desktop deployments.""" + +from __future__ import annotations + +from civiccore.platform.backup import ( + BackupItem, + BackupManifest, + BackupRestoreAction, + BackupRestorePlan, + BackupValidationResult, + build_backup_manifest, + plan_restore, + validate_backup_manifest, +) +from civiccore.platform.health import ( + PlatformHealthCheck, + PlatformHealthStatus, + PlatformHealthSummary, + summarize_platform_health, +) +from civiccore.platform.modules import ( + WINDOWS_LOCAL_BLOCKED_RUNTIME_KINDS, + ModuleBackupHook, + ModuleDependency, + ModuleHealthCheck, + ModuleManifest, + ModuleMigration, + ModuleModelRequirement, + ModulePermission, + ModuleRegistryEntry, + ModuleRegistryState, + ModuleRoute, + ModuleRuntimeRequirement, + ModuleService, + build_module_registry, + validate_windows_local_manifest, +) +from civiccore.platform.runtime import LocalRuntimeProfile, RuntimeActionResult +from civiccore.platform.task_queue import ( + LocalTask, + TaskHandler, + claim_next_local_task, + complete_local_task, + enqueue_local_task, + fail_local_task, + run_one_local_task, + task_row_to_envelope, +) +from civiccore.platform.tasks import ( + LocalTaskEnvelope, + LocalTaskResult, + TaskQueueSummary, + TaskRetryPolicy, + can_run_task, + next_retry_at, + record_task_attempt, + summarize_task_queue, +) + +__all__ = [ + "BackupItem", + "BackupManifest", + "BackupRestoreAction", + "BackupRestorePlan", + "BackupValidationResult", + "LocalRuntimeProfile", + "LocalTask", + "LocalTaskEnvelope", + "LocalTaskResult", + "ModuleBackupHook", + "ModuleDependency", + "ModuleHealthCheck", + "ModuleManifest", + "ModuleMigration", + "ModuleModelRequirement", + "ModulePermission", + "ModuleRegistryEntry", + "ModuleRegistryState", + "ModuleRoute", + "ModuleRuntimeRequirement", + "ModuleService", + "PlatformHealthCheck", + "PlatformHealthStatus", + "PlatformHealthSummary", + "RuntimeActionResult", + "TaskQueueSummary", + "TaskHandler", + "TaskRetryPolicy", + "WINDOWS_LOCAL_BLOCKED_RUNTIME_KINDS", + "build_backup_manifest", + "build_module_registry", + "can_run_task", + "claim_next_local_task", + "complete_local_task", + "enqueue_local_task", + "fail_local_task", + "next_retry_at", + "plan_restore", + "record_task_attempt", + "run_one_local_task", + "summarize_platform_health", + "summarize_task_queue", + "task_row_to_envelope", + "validate_backup_manifest", + "validate_windows_local_manifest", +] diff --git a/civiccore/platform/backup.py b/civiccore/platform/backup.py new file mode 100644 index 0000000..6e27da3 --- /dev/null +++ b/civiccore/platform/backup.py @@ -0,0 +1,209 @@ +"""Backup and restore manifest contracts for local CivicSuite installs.""" + +from __future__ import annotations + +import hashlib +from datetime import UTC, datetime +from pathlib import Path +from typing import Literal + +from pydantic import BaseModel, ConfigDict, Field + +RestoreActionStatus = Literal["ready", "missing", "hash_mismatch", "would_overwrite"] + + +class BackupItem(BaseModel): + """One file captured in a local backup.""" + + model_config = ConfigDict(extra="forbid") + + module_id: str = Field(pattern=r"^[a-z][a-z0-9-]{1,63}$") + relative_path: str = Field(min_length=1) + size_bytes: int = Field(ge=0) + sha256: str = Field(pattern=r"^[a-f0-9]{64}$") + required: bool = True + + +class BackupManifest(BaseModel): + """Versioned backup manifest for a CivicSuite local profile.""" + + model_config = ConfigDict(extra="forbid") + + schema_version: str = "1.0" + backup_id: str = Field(min_length=1) + city_profile_id: str = Field(min_length=1) + civiccore_version: str = Field(min_length=1) + created_at: datetime = Field(default_factory=lambda: datetime.now(UTC)) + modules: list[str] = Field(default_factory=list) + items: list[BackupItem] + + +class BackupValidationResult(BaseModel): + """Result of checking a backup manifest against files on disk.""" + + model_config = ConfigDict(extra="forbid") + + valid: bool + missing: list[str] = Field(default_factory=list) + hash_mismatches: list[str] = Field(default_factory=list) + + +class BackupRestoreAction(BaseModel): + """One file restore action and its safety status.""" + + model_config = ConfigDict(extra="forbid") + + source_relative_path: str + target_relative_path: str + status: RestoreActionStatus + message: str + + +class BackupRestorePlan(BaseModel): + """A non-destructive restore plan.""" + + model_config = ConfigDict(extra="forbid") + + ready: bool + actions: list[BackupRestoreAction] + blocked_reason: str | None = None + + +def build_backup_manifest( + *, + root: Path, + files: list[Path], + backup_id: str, + city_profile_id: str, + civiccore_version: str, + module_id: str, +) -> BackupManifest: + """Build a checksum manifest for files under a local data root.""" + + resolved_root = root.resolve() + items: list[BackupItem] = [] + for path in files: + resolved_path = path.resolve() + _assert_inside(resolved_path, resolved_root) + if not resolved_path.is_file(): + raise ValueError(f"backup item is not a file: {resolved_path}") + relative_path = resolved_path.relative_to(resolved_root).as_posix() + items.append( + BackupItem( + module_id=module_id, + relative_path=relative_path, + size_bytes=resolved_path.stat().st_size, + sha256=_sha256_file(resolved_path), + ) + ) + + return BackupManifest( + backup_id=backup_id, + city_profile_id=city_profile_id, + civiccore_version=civiccore_version, + modules=sorted({module_id for _item in items}), + items=sorted(items, key=lambda item: item.relative_path), + ) + + +def validate_backup_manifest(manifest: BackupManifest, *, root: Path) -> BackupValidationResult: + """Check that a manifest still matches a backup directory.""" + + resolved_root = root.resolve() + missing: list[str] = [] + mismatches: list[str] = [] + for item in manifest.items: + path = (resolved_root / item.relative_path).resolve() + _assert_inside(path, resolved_root) + if not path.is_file(): + missing.append(item.relative_path) + continue + if _sha256_file(path) != item.sha256: + mismatches.append(item.relative_path) + + return BackupValidationResult( + valid=not missing and not mismatches, + missing=missing, + hash_mismatches=mismatches, + ) + + +def plan_restore( + manifest: BackupManifest, + *, + backup_root: Path, + restore_root: Path, + overwrite: bool = False, +) -> BackupRestorePlan: + """Return a non-destructive restore plan for a backup manifest.""" + + resolved_backup_root = backup_root.resolve() + resolved_restore_root = restore_root.resolve() + actions: list[BackupRestoreAction] = [] + + for item in manifest.items: + source = (resolved_backup_root / item.relative_path).resolve() + target = (resolved_restore_root / item.relative_path).resolve() + _assert_inside(source, resolved_backup_root) + _assert_inside(target, resolved_restore_root) + + if not source.is_file(): + actions.append( + BackupRestoreAction( + source_relative_path=item.relative_path, + target_relative_path=item.relative_path, + status="missing", + message="Backup file is missing.", + ) + ) + continue + if _sha256_file(source) != item.sha256: + actions.append( + BackupRestoreAction( + source_relative_path=item.relative_path, + target_relative_path=item.relative_path, + status="hash_mismatch", + message="Backup file checksum does not match the manifest.", + ) + ) + continue + if target.exists() and not overwrite: + actions.append( + BackupRestoreAction( + source_relative_path=item.relative_path, + target_relative_path=item.relative_path, + status="would_overwrite", + message="Restore target already exists.", + ) + ) + continue + actions.append( + BackupRestoreAction( + source_relative_path=item.relative_path, + target_relative_path=item.relative_path, + status="ready", + message="Ready to restore.", + ) + ) + + blockers = [action for action in actions if action.status != "ready"] + return BackupRestorePlan( + ready=not blockers, + actions=actions, + blocked_reason=None if not blockers else "Restore plan has file safety blockers.", + ) + + +def _assert_inside(path: Path, root: Path) -> None: + try: + path.relative_to(root) + except ValueError as exc: + raise ValueError(f"path is outside the expected root: {path}") from exc + + +def _sha256_file(path: Path) -> str: + hasher = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + hasher.update(chunk) + return hasher.hexdigest() diff --git a/civiccore/platform/health.py b/civiccore/platform/health.py new file mode 100644 index 0000000..d6fe659 --- /dev/null +++ b/civiccore/platform/health.py @@ -0,0 +1,104 @@ +"""Shared platform health projection for local CivicSuite surfaces.""" + +from __future__ import annotations + +from datetime import UTC, datetime +from typing import Literal + +from pydantic import BaseModel, ConfigDict, Field + +PlatformHealthStatus = Literal["ok", "needs_setup", "degraded", "blocked"] +_STATUS_RANK: dict[PlatformHealthStatus, int] = { + "ok": 0, + "needs_setup": 1, + "degraded": 2, + "blocked": 3, +} + + +class PlatformHealthCheck(BaseModel): + """One health check result suitable for staff/admin health screens.""" + + model_config = ConfigDict(extra="forbid") + + component: str = Field(min_length=1) + label: str = Field(min_length=1) + status: PlatformHealthStatus + message: str = Field(min_length=1) + next_action: str = Field(min_length=1) + blocking: bool = False + checked_at: datetime = Field(default_factory=lambda: datetime.now(UTC)) + admin_detail: str | None = Field(default=None, min_length=1) + + +class PlatformHealthSummary(BaseModel): + """Aggregated health status for the umbrella shell.""" + + model_config = ConfigDict(extra="forbid") + + status: PlatformHealthStatus + checks: list[PlatformHealthCheck] + next_action: str + staff_message: str + admin_message: str + blocked_components: list[str] + degraded_components: list[str] + + @property + def is_release_blocked(self) -> bool: + return self.status == "blocked" or bool(self.blocked_components) + + +def summarize_platform_health(checks: list[PlatformHealthCheck]) -> PlatformHealthSummary: + """Return the worst health status plus plain-English action copy.""" + + if not checks: + return PlatformHealthSummary( + status="needs_setup", + checks=[], + next_action="Run first-run setup so CivicSuite can verify the local services.", + staff_message="CivicSuite has not completed setup yet.", + admin_message="No platform health checks have reported.", + blocked_components=[], + degraded_components=[], + ) + + worst = max((check.status for check in checks), key=lambda status: _STATUS_RANK[status]) + action_source = next( + ( + check + for check in checks + if check.blocking or check.status in {"blocked", "needs_setup", "degraded"} + ), + checks[0], + ) + blocked = [ + check.component for check in checks if check.blocking or check.status == "blocked" + ] + degraded = [check.component for check in checks if check.status == "degraded"] + + if blocked: + staff_message = "CivicSuite needs attention before affected workflows can continue." + elif worst == "degraded": + staff_message = "CivicSuite is running, but one or more background checks need attention." + elif worst == "needs_setup": + staff_message = "CivicSuite needs setup before all workflows are available." + else: + staff_message = "CivicSuite is healthy." + + return PlatformHealthSummary( + status=worst, + checks=checks, + next_action=action_source.next_action, + staff_message=staff_message, + admin_message=_admin_message(checks), + blocked_components=blocked, + degraded_components=degraded, + ) + + +def _admin_message(checks: list[PlatformHealthCheck]) -> str: + details = [check.admin_detail for check in checks if check.admin_detail] + if details: + return " ".join(details) + return "All reported checks are available in the local health center." diff --git a/civiccore/platform/modules.py b/civiccore/platform/modules.py new file mode 100644 index 0000000..9610617 --- /dev/null +++ b/civiccore/platform/modules.py @@ -0,0 +1,368 @@ +"""Module manifest and registry contracts for local CivicSuite deployments.""" + +from __future__ import annotations + +from typing import Any, Literal + +from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator + +ModuleSurface = Literal["staff", "resident", "admin", "system"] +InstallProfile = Literal["windows_local", "server", "developer"] +ServiceType = Literal[ + "python-service", + "postgres-schema", + "file-store", + "ollama-model", + "tauri-command", + "local-api", + "background-worker", +] +RuntimeRequirementKind = Literal[ + "civiccore", + "postgres", + "pgvector", + "python", + "file_storage", + "ollama", + "llm_model", + "tauri_command", + "local_api", + "windows_service", + "webview2", + "external_connector", + "docker", + "wsl", + "linux_shell", + "terminal", +] +ModuleRegistryStatus = Literal["enabled", "disabled", "blocked"] + +MODULE_ID_PATTERN = r"^[a-z][a-z0-9-]{1,63}$" +SLUG_PATTERN = r"^[a-z][a-z0-9_-]{1,63}$" +WINDOWS_LOCAL_BLOCKED_RUNTIME_KINDS = frozenset({"docker", "wsl", "linux_shell", "terminal"}) + + +class ModuleDependency(BaseModel): + """Another CivicSuite module required by a manifest.""" + + model_config = ConfigDict(extra="forbid") + + module_id: str = Field(pattern=MODULE_ID_PATTERN) + min_version: str | None = Field(default=None, min_length=1) + required: bool = True + + +class ModulePermission(BaseModel): + """Permission surfaced to the shell and module manager.""" + + model_config = ConfigDict(extra="forbid") + + key: str = Field(pattern=SLUG_PATTERN) + label: str = Field(min_length=1) + description: str = Field(min_length=1) + surface: ModuleSurface = "staff" + roles: list[str] = Field(default_factory=list) + + @field_validator("roles") + @classmethod + def validate_roles(cls, value: list[str]) -> list[str]: + if not value: + raise ValueError("roles must name at least one role that grants the permission") + return value + + +class ModuleRoute(BaseModel): + """A route the desktop shell may expose for a module.""" + + model_config = ConfigDict(extra="forbid") + + id: str = Field(pattern=SLUG_PATTERN) + path: str = Field(pattern=r"^/[a-z0-9/_:-]*$") + label: str = Field(min_length=1) + surface: ModuleSurface = "staff" + permission: str | None = Field(default=None, pattern=SLUG_PATTERN) + + +class ModuleService(BaseModel): + """A local service, schema, or command needed by a module.""" + + model_config = ConfigDict(extra="forbid") + + id: str = Field(pattern=SLUG_PATTERN) + service_type: ServiceType + description: str = Field(min_length=1) + required: bool = True + start_order: int = Field(default=100, ge=0) + health_check_id: str | None = Field(default=None, pattern=SLUG_PATTERN) + + +class ModuleMigration(BaseModel): + """A migration contract that the installer/runtime must apply or verify.""" + + model_config = ConfigDict(extra="forbid") + + id: str = Field(pattern=SLUG_PATTERN) + description: str = Field(min_length=1) + owner_schema: str | None = Field(default=None, min_length=1) + required: bool = True + + +class ModuleHealthCheck(BaseModel): + """A module health check known to the umbrella health center.""" + + model_config = ConfigDict(extra="forbid") + + id: str = Field(pattern=SLUG_PATTERN) + label: str = Field(min_length=1) + description: str = Field(min_length=1) + blocking: bool = True + surface: Literal["staff", "admin", "system"] = "admin" + repair_action: str | None = Field(default=None, min_length=1) + + +class ModuleBackupHook(BaseModel): + """A backup/restore hook the module contributes to the shared backup plan.""" + + model_config = ConfigDict(extra="forbid") + + id: str = Field(pattern=SLUG_PATTERN) + label: str = Field(min_length=1) + includes: list[str] = Field(min_length=1) + restore_supported: bool = True + + +class ModuleModelRequirement(BaseModel): + """A model requirement declared by a module.""" + + model_config = ConfigDict(extra="forbid") + + id: str = Field(pattern=SLUG_PATTERN) + provider: Literal["ollama", "local", "none", "openai", "anthropic"] = "ollama" + model_name: str = Field(min_length=1) + required: bool = True + local_only: bool = True + minimum_context_tokens: int | None = Field(default=None, ge=1) + checksum_sha256: str | None = Field(default=None, pattern=r"^[a-fA-F0-9]{64}$") + download_url: str | None = Field(default=None, min_length=1) + license_name: str | None = Field(default=None, min_length=1) + + +class ModuleRuntimeRequirement(BaseModel): + """Runtime capability needed by a module.""" + + model_config = ConfigDict(extra="forbid") + + kind: RuntimeRequirementKind + name: str = Field(min_length=1) + description: str = Field(min_length=1) + required: bool = True + operator_visible: bool = False + + +class ModuleManifest(BaseModel): + """Complete manifest for a CivicSuite module package.""" + + model_config = ConfigDict(extra="forbid") + + module_id: str = Field(pattern=MODULE_ID_PATTERN) + name: str = Field(min_length=1) + version: str = Field(min_length=1) + package_name: str = Field(min_length=1) + civiccore_min_version: str = Field(min_length=1) + enabled_by_default: bool = True + surfaces: list[ModuleSurface] = Field(default_factory=lambda: ["staff"]) + install_profiles: list[InstallProfile] = Field(default_factory=lambda: ["windows_local"]) + dependencies: list[ModuleDependency] = Field(default_factory=list) + routes: list[ModuleRoute] = Field(default_factory=list) + permissions: list[ModulePermission] = Field(default_factory=list) + services: list[ModuleService] = Field(default_factory=list) + migrations: list[ModuleMigration] = Field(default_factory=list) + health_checks: list[ModuleHealthCheck] = Field(default_factory=list) + backup_hooks: list[ModuleBackupHook] = Field(default_factory=list) + model_requirements: list[ModuleModelRequirement] = Field(default_factory=list) + runtime_requirements: list[ModuleRuntimeRequirement] = Field(default_factory=list) + settings_schema_version: str | None = Field(default=None, min_length=1) + metadata: dict[str, Any] = Field(default_factory=dict) + + @field_validator("surfaces", "install_profiles") + @classmethod + def validate_non_empty_list(cls, value: list[str]) -> list[str]: + if not value: + raise ValueError("list must contain at least one item") + return value + + @model_validator(mode="after") + def validate_cross_references(self) -> ModuleManifest: + _assert_unique("permission", [item.key for item in self.permissions]) + _assert_unique("route", [item.id for item in self.routes]) + _assert_unique("service", [item.id for item in self.services]) + _assert_unique("migration", [item.id for item in self.migrations]) + _assert_unique("health_check", [item.id for item in self.health_checks]) + _assert_unique("backup_hook", [item.id for item in self.backup_hooks]) + _assert_unique("model_requirement", [item.id for item in self.model_requirements]) + + permission_keys = {item.key for item in self.permissions} + for route in self.routes: + if route.permission and route.permission not in permission_keys: + raise ValueError( + f"route {route.id!r} references unknown permission {route.permission!r}" + ) + + health_check_ids = {item.id for item in self.health_checks} + for service in self.services: + if service.health_check_id and service.health_check_id not in health_check_ids: + raise ValueError( + f"service {service.id!r} references unknown health check " + f"{service.health_check_id!r}" + ) + + for dependency in self.dependencies: + if dependency.module_id == self.module_id: + raise ValueError("module cannot depend on itself") + + if "windows_local" in self.install_profiles: + _validate_windows_runtime_requirements(self) + + return self + + +class ModuleRegistryEntry(BaseModel): + """One module's resolved state in a runtime profile.""" + + model_config = ConfigDict(extra="forbid") + + module_id: str = Field(pattern=MODULE_ID_PATTERN) + name: str = Field(min_length=1) + version: str = Field(min_length=1) + enabled: bool + locked: bool = False + status: ModuleRegistryStatus + reason: str | None = None + manifest: ModuleManifest + + +class ModuleRegistryState(BaseModel): + """Resolved module registry for the desktop shell and installer.""" + + model_config = ConfigDict(extra="forbid") + + profile: InstallProfile = "windows_local" + civiccore_version: str = Field(min_length=1) + civiccore_locked: bool = True + entries: list[ModuleRegistryEntry] + + @property + def enabled_module_ids(self) -> list[str]: + return [entry.module_id for entry in self.entries if entry.enabled] + + @property + def blocked_module_ids(self) -> list[str]: + return [entry.module_id for entry in self.entries if entry.status == "blocked"] + + +def validate_windows_local_manifest(manifest: ModuleManifest | dict[str, Any]) -> ModuleManifest: + """Validate a manifest for the Windows desktop profile.""" + + parsed = manifest if isinstance(manifest, ModuleManifest) else ModuleManifest.model_validate(manifest) + if "windows_local" not in parsed.install_profiles: + raise ValueError(f"{parsed.module_id} does not declare support for windows_local") + _validate_windows_runtime_requirements(parsed) + return parsed + + +def build_module_registry( + manifests: list[ModuleManifest | dict[str, Any]], + *, + civiccore_version: str, + selected_module_ids: set[str] | None = None, + profile: InstallProfile = "windows_local", +) -> ModuleRegistryState: + """Resolve manifests into an installer/runtime registry state.""" + + parsed = [ + item if isinstance(item, ModuleManifest) else ModuleManifest.model_validate(item) + for item in manifests + ] + ids = [item.module_id for item in parsed] + _assert_unique("module", ids) + + by_id = {item.module_id: item for item in parsed} + selected = ( + set(selected_module_ids) + if selected_module_ids is not None + else {item.module_id for item in parsed if item.enabled_by_default} + ) + unknown = selected - set(by_id) + if unknown: + raise ValueError(f"selected modules are not installed: {sorted(unknown)}") + + entries: list[ModuleRegistryEntry] = [] + for manifest in parsed: + enabled = manifest.module_id in selected + status: ModuleRegistryStatus = "enabled" if enabled else "disabled" + reason: str | None = None + + if profile not in manifest.install_profiles: + enabled = False + status = "blocked" + reason = f"{manifest.name} is not available for the {profile} profile." + elif enabled: + missing = [ + dependency.module_id + for dependency in manifest.dependencies + if dependency.required and dependency.module_id not in selected + ] + if missing: + enabled = False + status = "blocked" + reason = f"Missing required module dependencies: {', '.join(sorted(missing))}." + + entries.append( + ModuleRegistryEntry( + module_id=manifest.module_id, + name=manifest.name, + version=manifest.version, + enabled=enabled, + locked=False, + status=status, + reason=reason, + manifest=manifest, + ) + ) + + return ModuleRegistryState( + profile=profile, + civiccore_version=civiccore_version, + civiccore_locked=True, + entries=sorted(entries, key=lambda entry: entry.module_id), + ) + + +def _assert_unique(label: str, values: list[str]) -> None: + seen: set[str] = set() + duplicates = sorted({value for value in values if value in seen or seen.add(value)}) + if duplicates: + raise ValueError(f"duplicate {label} ids: {', '.join(duplicates)}") + + +def _validate_windows_runtime_requirements(manifest: ModuleManifest) -> None: + blocked = [ + requirement.kind + for requirement in manifest.runtime_requirements + if requirement.required and requirement.kind in WINDOWS_LOCAL_BLOCKED_RUNTIME_KINDS + ] + if blocked: + blocked_list = ", ".join(sorted(set(blocked))) + raise ValueError( + f"{manifest.module_id} cannot require {blocked_list} for the Windows local profile" + ) + terminal_only = [ + requirement.name + for requirement in manifest.runtime_requirements + if requirement.required and requirement.operator_visible and requirement.kind == "terminal" + ] + if terminal_only: + raise ValueError( + f"{manifest.module_id} exposes terminal-only operator setup: " + f"{', '.join(sorted(terminal_only))}" + ) diff --git a/civiccore/platform/runtime.py b/civiccore/platform/runtime.py new file mode 100644 index 0000000..24d7dfd --- /dev/null +++ b/civiccore/platform/runtime.py @@ -0,0 +1,63 @@ +"""Runtime action contracts for the Windows local desktop shell.""" + +from __future__ import annotations + +from datetime import UTC, datetime +from pathlib import Path +from typing import Literal + +from pydantic import BaseModel, ConfigDict, Field, field_validator + +from civiccore.platform.modules import ModuleRegistryState + +RuntimeAction = Literal[ + "install", + "first_run", + "health_check", + "repair", + "backup", + "restore", + "uninstall", + "module_install", + "module_disable", + "model_download", +] +RuntimeActionStatus = Literal["pending", "running", "succeeded", "needs_action", "failed", "blocked"] + + +class RuntimeActionResult(BaseModel): + """Plain-English result returned by installer/runtime APIs.""" + + model_config = ConfigDict(extra="forbid") + + action: RuntimeAction + status: RuntimeActionStatus + title: str = Field(min_length=1) + message: str = Field(min_length=1) + next_action: str | None = Field(default=None, min_length=1) + evidence: dict[str, str | int | bool | None] = Field(default_factory=dict) + started_at: datetime = Field(default_factory=lambda: datetime.now(UTC)) + completed_at: datetime | None = None + + +class LocalRuntimeProfile(BaseModel): + """Resolved local runtime profile for one installed city.""" + + model_config = ConfigDict(extra="forbid") + + profile_id: str = Field(min_length=1) + city_name: str = Field(min_length=1) + data_root: Path + backup_root: Path + module_registry: ModuleRegistryState + model_provider: str = "ollama" + local_only: bool = True + network_allowed_for: list[str] = Field(default_factory=list) + + @field_validator("network_allowed_for") + @classmethod + def validate_network_reasons(cls, value: list[str]) -> list[str]: + for reason in value: + if not reason.strip(): + raise ValueError("network_allowed_for entries cannot be blank") + return value diff --git a/civiccore/platform/task_queue.py b/civiccore/platform/task_queue.py new file mode 100644 index 0000000..24f6fc1 --- /dev/null +++ b/civiccore/platform/task_queue.py @@ -0,0 +1,211 @@ +"""PostgreSQL-backed local task queue for CivicSuite desktop runtimes.""" + +from __future__ import annotations + +import inspect +from collections.abc import Awaitable, Callable +from datetime import UTC, datetime +from typing import Any + +from sqlalchemy import DateTime, Integer, String, Text, select +from sqlalchemy.dialects.postgresql import JSONB +from sqlalchemy.ext.asyncio import AsyncSession +from sqlalchemy.orm import Mapped, mapped_column + +from civiccore.db import Base +from civiccore.platform.tasks import ( + LocalTaskEnvelope, + TaskRetryPolicy, + record_task_attempt, +) + +TaskHandler = Callable[[LocalTaskEnvelope], object | Awaitable[object]] + + +class LocalTask(Base): + """Durable local task row used by the Windows local runtime.""" + + __tablename__ = "civiccore_local_tasks" + + task_id: Mapped[str] = mapped_column(String(128), primary_key=True) + module_id: Mapped[str] = mapped_column(String(64), index=True) + task_type: Mapped[str] = mapped_column(String(100), index=True) + status: Mapped[str] = mapped_column(String(20), index=True, default="queued") + payload: Mapped[dict[str, Any]] = mapped_column(JSONB, default=dict) + attempt_count: Mapped[int] = mapped_column(Integer, default=0) + queued_at: Mapped[datetime] = mapped_column(DateTime(timezone=True)) + available_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True) + last_error: Mapped[str | None] = mapped_column(Text, nullable=True) + idempotency_key: Mapped[str | None] = mapped_column(String(255), nullable=True, unique=True) + audit_subject_id: Mapped[str | None] = mapped_column(String(255), nullable=True) + + +def task_row_to_envelope(row: LocalTask) -> LocalTaskEnvelope: + """Convert an ORM row into the public task contract.""" + + return LocalTaskEnvelope( + task_id=row.task_id, + module_id=row.module_id, + task_type=row.task_type, + status=row.status, # type: ignore[arg-type] + payload=row.payload or {}, + attempt_count=row.attempt_count, + queued_at=row.queued_at, + available_at=row.available_at, + last_error=row.last_error, + idempotency_key=row.idempotency_key, + audit_subject_id=row.audit_subject_id, + ) + + +async def enqueue_local_task( + session: AsyncSession, + task: LocalTaskEnvelope, +) -> LocalTaskEnvelope: + """Insert a task unless its idempotency key already exists.""" + + if task.idempotency_key: + existing = await session.scalar( + select(LocalTask).where(LocalTask.idempotency_key == task.idempotency_key) + ) + if existing is not None: + return task_row_to_envelope(existing) + + row = LocalTask( + task_id=task.task_id, + module_id=task.module_id, + task_type=task.task_type, + status=task.status, + payload=task.payload, + attempt_count=task.attempt_count, + queued_at=task.queued_at, + available_at=task.available_at, + last_error=task.last_error, + idempotency_key=task.idempotency_key, + audit_subject_id=task.audit_subject_id, + ) + session.add(row) + await session.flush() + return task_row_to_envelope(row) + + +async def claim_next_local_task( + session: AsyncSession, + *, + module_id: str | None = None, + now: datetime | None = None, +) -> LocalTaskEnvelope | None: + """Claim the next queued or due retry task.""" + + current_time = now or datetime.now(UTC) + query = ( + select(LocalTask) + .where( + (LocalTask.status == "queued") + | ((LocalTask.status == "retry_wait") & (LocalTask.available_at <= current_time)) + ) + .order_by(LocalTask.queued_at.asc()) + .with_for_update(skip_locked=True) + .limit(1) + ) + if module_id is not None: + query = query.where(LocalTask.module_id == module_id) + + row = await session.scalar(query) + if row is None: + return None + + row.status = "running" + row.available_at = None + await session.flush() + return task_row_to_envelope(row) + + +async def complete_local_task(session: AsyncSession, task_id: str) -> LocalTaskEnvelope: + """Mark a claimed task as succeeded.""" + + row = await _get_task_row(session, task_id) + row.status = "succeeded" + row.last_error = None + row.available_at = None + await session.flush() + return task_row_to_envelope(row) + + +async def fail_local_task( + session: AsyncSession, + task_id: str, + *, + error: str, + now: datetime | None = None, + policy: TaskRetryPolicy | None = None, +) -> LocalTaskEnvelope: + """Record a failed attempt and either schedule retry or exhaust the task.""" + + row = await _get_task_row(session, task_id) + result = record_task_attempt( + task_row_to_envelope(row), + success=False, + error=error, + now=now, + policy=policy, + ) + _apply_envelope(row, result.task) + await session.flush() + return task_row_to_envelope(row) + + +async def run_one_local_task( + session: AsyncSession, + *, + handlers: dict[str, TaskHandler], + module_id: str | None = None, + now: datetime | None = None, + retry_policy: TaskRetryPolicy | None = None, +) -> LocalTaskEnvelope | None: + """Claim and run one task with a registered in-process handler.""" + + task = await claim_next_local_task(session, module_id=module_id, now=now) + if task is None: + return None + + handler = handlers.get(task.task_type) + if handler is None: + return await fail_local_task( + session, + task.task_id, + error=f"No handler registered for task type {task.task_type!r}.", + now=now, + policy=retry_policy, + ) + + try: + result = handler(task) + if inspect.isawaitable(result): + await result + except Exception as exc: # pragma: no cover - exact handler failures are module-owned + return await fail_local_task( + session, + task.task_id, + error=str(exc), + now=now, + policy=retry_policy, + ) + return await complete_local_task(session, task.task_id) + + +async def _get_task_row(session: AsyncSession, task_id: str) -> LocalTask: + row = await session.get(LocalTask, task_id) + if row is None: + raise LookupError(f"local task not found: {task_id}") + return row + + +def _apply_envelope(row: LocalTask, envelope: LocalTaskEnvelope) -> None: + row.status = envelope.status + row.payload = envelope.payload + row.attempt_count = envelope.attempt_count + row.available_at = envelope.available_at + row.last_error = envelope.last_error + row.idempotency_key = envelope.idempotency_key + row.audit_subject_id = envelope.audit_subject_id diff --git a/civiccore/platform/tasks.py b/civiccore/platform/tasks.py new file mode 100644 index 0000000..dfa5de7 --- /dev/null +++ b/civiccore/platform/tasks.py @@ -0,0 +1,166 @@ +"""Storage-neutral local task contracts for CivicSuite modules.""" + +from __future__ import annotations + +from collections import Counter +from datetime import UTC, datetime, timedelta +from typing import Any, Literal + +from pydantic import BaseModel, ConfigDict, Field, model_validator + +TaskStatus = Literal["queued", "running", "succeeded", "retry_wait", "failed", "cancelled"] + + +class TaskRetryPolicy(BaseModel): + """Retry settings for local durable task queues.""" + + model_config = ConfigDict(extra="forbid") + + max_attempts: int = Field(default=3, ge=1) + base_delay_seconds: int = Field(default=30, ge=0) + max_delay_seconds: int = Field(default=3600, ge=0) + + +class LocalTaskEnvelope(BaseModel): + """Storage-neutral task record for module-owned persistence.""" + + model_config = ConfigDict(extra="forbid") + + task_id: str = Field(min_length=1) + module_id: str = Field(pattern=r"^[a-z][a-z0-9-]{1,63}$") + task_type: str = Field(min_length=1) + status: TaskStatus = "queued" + payload: dict[str, Any] = Field(default_factory=dict) + attempt_count: int = Field(default=0, ge=0) + queued_at: datetime = Field(default_factory=lambda: datetime.now(UTC)) + available_at: datetime | None = None + last_error: str | None = Field(default=None, min_length=1) + idempotency_key: str | None = Field(default=None, min_length=1) + audit_subject_id: str | None = Field(default=None, min_length=1) + + @model_validator(mode="after") + def validate_retry_state(self) -> LocalTaskEnvelope: + if self.status == "retry_wait" and self.available_at is None: + raise ValueError("retry_wait tasks must include available_at") + if self.status == "failed" and not self.last_error: + raise ValueError("failed tasks must include last_error") + return self + + +class LocalTaskResult(BaseModel): + """Result returned after recording a local task attempt.""" + + model_config = ConfigDict(extra="forbid") + + task: LocalTaskEnvelope + message: str + retry_at: datetime | None = None + + +class TaskQueueSummary(BaseModel): + """Counts and operator copy for a module task queue.""" + + model_config = ConfigDict(extra="forbid") + + total: int + counts: dict[str, int] + blocked: bool + message: str + + +def next_retry_at( + attempt_count: int, + failed_at: datetime, + *, + policy: TaskRetryPolicy | None = None, +) -> datetime | None: + """Return the next retry time after a failed attempt, or None when exhausted.""" + + active_policy = policy or TaskRetryPolicy() + if attempt_count >= active_policy.max_attempts: + return None + delay = active_policy.base_delay_seconds * (2 ** max(0, attempt_count - 1)) + delay = min(delay, active_policy.max_delay_seconds) + return failed_at + timedelta(seconds=delay) + + +def can_run_task(task: LocalTaskEnvelope, *, now: datetime | None = None) -> bool: + """Return True when a queued or retry-wait task can be claimed.""" + + current_time = now or datetime.now(UTC) + if task.status == "queued": + return True + if task.status == "retry_wait" and task.available_at is not None: + return task.available_at <= current_time + return False + + +def record_task_attempt( + task: LocalTaskEnvelope, + *, + success: bool, + now: datetime | None = None, + error: str | None = None, + policy: TaskRetryPolicy | None = None, +) -> LocalTaskResult: + """Return an updated task envelope after one local worker attempt.""" + + current_time = now or datetime.now(UTC) + attempt_count = task.attempt_count + 1 + if success: + updated = task.model_copy( + update={ + "status": "succeeded", + "attempt_count": attempt_count, + "available_at": None, + "last_error": None, + } + ) + return LocalTaskResult(task=updated, message="Task completed successfully.") + + if not error: + raise ValueError("error is required when recording a failed task attempt") + + retry_at = next_retry_at(attempt_count, current_time, policy=policy) + if retry_at is None: + updated = task.model_copy( + update={ + "status": "failed", + "attempt_count": attempt_count, + "available_at": None, + "last_error": error, + } + ) + return LocalTaskResult(task=updated, message="Task failed and retries are exhausted.") + + updated = task.model_copy( + update={ + "status": "retry_wait", + "attempt_count": attempt_count, + "available_at": retry_at, + "last_error": error, + } + ) + return LocalTaskResult(task=updated, message="Task failed and is waiting to retry.", retry_at=retry_at) + + +def summarize_task_queue(tasks: list[LocalTaskEnvelope]) -> TaskQueueSummary: + """Return counts and operator copy for a queue view.""" + + counts = Counter(task.status for task in tasks) + blocked = bool(counts.get("failed")) + if blocked: + message = "One or more local tasks failed and need review." + elif counts.get("retry_wait"): + message = "Some local tasks are waiting to retry." + elif counts.get("running") or counts.get("queued"): + message = "Local tasks are active." + else: + message = "No local task issues are reported." + + return TaskQueueSummary( + total=len(tasks), + counts=dict(sorted(counts.items())), + blocked=blocked, + message=message, + ) diff --git a/civiccore/tasks/__init__.py b/civiccore/tasks/__init__.py new file mode 100644 index 0000000..547de9a --- /dev/null +++ b/civiccore/tasks/__init__.py @@ -0,0 +1,7 @@ +"""Local task worker entry points for CivicCore.""" + +from __future__ import annotations + +from civiccore.tasks.registry import get_task_handlers, register_task_handler + +__all__ = ["get_task_handlers", "register_task_handler"] diff --git a/civiccore/tasks/registry.py b/civiccore/tasks/registry.py new file mode 100644 index 0000000..09b3a92 --- /dev/null +++ b/civiccore/tasks/registry.py @@ -0,0 +1,28 @@ +"""In-process task handler registry for local CivicSuite workers.""" + +from __future__ import annotations + +from collections.abc import Callable + +from civiccore.platform.task_queue import TaskHandler + +_HANDLERS: dict[str, TaskHandler] = {} + + +def register_task_handler(task_type: str) -> Callable[[TaskHandler], TaskHandler]: + """Register a callable task handler by task type.""" + + if not task_type.strip(): + raise ValueError("task_type cannot be blank") + + def decorator(handler: TaskHandler) -> TaskHandler: + _HANDLERS[task_type] = handler + return handler + + return decorator + + +def get_task_handlers() -> dict[str, TaskHandler]: + """Return a copy of the registered task handlers.""" + + return dict(_HANDLERS) diff --git a/civiccore/tasks/worker.py b/civiccore/tasks/worker.py new file mode 100644 index 0000000..189e2d6 --- /dev/null +++ b/civiccore/tasks/worker.py @@ -0,0 +1,78 @@ +"""CLI worker for the CivicCore PostgreSQL-backed local task queue.""" + +from __future__ import annotations + +import argparse +import asyncio +import importlib +import os +from collections.abc import Sequence + +from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine + +from civiccore.platform.task_queue import run_one_local_task +from civiccore.platform.tasks import TaskRetryPolicy +from civiccore.tasks.registry import get_task_handlers + + +def _async_database_url(url: str) -> str: + if "+psycopg2" in url: + return url.replace("postgresql+psycopg2", "postgresql+asyncpg") + if url.startswith("postgresql://"): + return url.replace("postgresql://", "postgresql+asyncpg://", 1) + return url + + +def _load_handler_modules(raw_modules: str | None) -> None: + if not raw_modules: + return + for module_name in [item.strip() for item in raw_modules.split(",") if item.strip()]: + importlib.import_module(module_name) + + +async def _run_worker(args: argparse.Namespace) -> int: + database_url = os.environ.get("DATABASE_URL") + if not database_url: + raise RuntimeError("DATABASE_URL is required for civiccore.tasks.worker") + _load_handler_modules(os.environ.get("CIVICCORE_TASK_HANDLER_MODULES")) + handlers = get_task_handlers() + + engine = create_async_engine(_async_database_url(database_url)) + session_factory = async_sessionmaker(engine, expire_on_commit=False) + retry_policy = TaskRetryPolicy( + max_attempts=args.max_attempts, + base_delay_seconds=args.base_delay_seconds, + max_delay_seconds=args.max_delay_seconds, + ) + + try: + while True: + async with session_factory() as session: + task = await run_one_local_task( + session, + handlers=handlers, + retry_policy=retry_policy, + ) + await session.commit() + if args.once: + return 0 + if task is None: + await asyncio.sleep(args.poll_seconds) + finally: + await engine.dispose() + + +def main(argv: Sequence[str] | None = None) -> int: + parser = argparse.ArgumentParser(description="Run the CivicCore local task worker.") + parser.add_argument("--backend", choices=["postgres"], default="postgres") + parser.add_argument("--once", action="store_true") + parser.add_argument("--poll-seconds", type=float, default=2.0) + parser.add_argument("--max-attempts", type=int, default=3) + parser.add_argument("--base-delay-seconds", type=int, default=30) + parser.add_argument("--max-delay-seconds", type=int, default=3600) + args = parser.parse_args(argv) + return asyncio.run(_run_worker(args)) + + +if __name__ == "__main__": # pragma: no cover - exercised through CLI smoke + raise SystemExit(main()) diff --git a/docs/audits/audit-lite-windows-local-platform-contracts-2026-06-13.md b/docs/audits/audit-lite-windows-local-platform-contracts-2026-06-13.md new file mode 100644 index 0000000..1952b79 --- /dev/null +++ b/docs/audits/audit-lite-windows-local-platform-contracts-2026-06-13.md @@ -0,0 +1,33 @@ +# Audit Lite: Windows-Local Platform Contracts + +Date: 2026-06-13 +Repo: CivicSuite/civiccore +Branch: work/windows-local-platform-contracts +Scope: `civiccore.platform` module manifest, health, task, backup/restore, runtime contracts, and PostgreSQL-backed local task queue. + +## Verdict + +Unresolved findings: 0 Blocker / 0 Critical / 0 Major / 0 Minor / 0 Nit + +This slice is acceptable to push. It adds real importable CivicCore contracts and a launchable local task worker for the Windows-local desktop path without claiming downstream module wiring is complete. + +## Evidence + +- `python -m pip install -e .[dev]` passed after installing the repo-declared dev extras. +- `python -m pytest` passed: 302 passed. +- `python -m ruff check civiccore tests` passed. +- `python -m build` passed and included `civiccore/platform` in the sdist and wheel. +- `git diff --check` passed. +- Focused platform/public API smoke passed: `tests/test_platform_contracts.py`, `tests/test_public_api_v03.py`, and `tests/test_smoke.py`. + +## Five-Lens Review + +- Engineering: Pass. New contracts validate the Windows-local profile against required Docker, WSL, Linux shell, and terminal requirements. The task queue now has a real migration, ORM row, async queue helpers, handler runner, and worker CLI. +- UX: Pass. Health and runtime result contracts require plain-English messages and next actions for clerk/admin surfaces. +- Tests: Pass. Tests cover positive registry resolution, blocked dependency state, forbidden runtime requirements, local task retry/exhaustion, PostgreSQL-backed enqueue/claim/complete/fail/worker behavior, backup checksum validation, restore overwrite blocking, and local-first runtime profile defaults. +- Docs: Pass. README and package metadata now describe the shipped platform contracts and distinguish contracts from worker/runtime execution. +- QA: Pass. Public package root exports were updated and covered by the public API smoke tests; packaging confirmed the new package is included. + +## Notes + +The configured `audit-lite` skill file was not present at `C:\Users\scott\.codex\skills\audit-lite\SKILL.md` in this session, so this report follows the in-repo five-lens self-audit format as the fallback. Downstream CivicRecords, CivicClerk, CivicCode, and desktop-shell adoption of these contracts and task handlers remains in subsequent slices. diff --git a/docs/diagrams/civiccore-extraction-map.mmd b/docs/diagrams/civiccore-extraction-map.mmd index 40ad2a2..608e5f5 100644 --- a/docs/diagrams/civiccore-extraction-map.mmd +++ b/docs/diagrams/civiccore-extraction-map.mmd @@ -1,6 +1,6 @@ graph TB - subgraph SHIPPED["civiccore v0.2.0 — SHIPPING TODAY"] - Mig["civiccore.migrations
runner + idempotent guards
civiccore_0001_baseline_v1
civiccore_0002_llm"] + subgraph SHIPPED["civiccore v1.2.0 - shipped core surface"] + Mig["civiccore.migrations
runner + idempotent guards
civiccore_0001_baseline_v1
civiccore_0002_llm
civiccore_0003_local_task_queue"] DB["civiccore.db.Base
shared SQLAlchemy
declarative base"] LLM["civiccore.llm
providers + templates +
registry + context +
structured output"] end diff --git a/docs/diagrams/civiccore-extraction-map.png b/docs/diagrams/civiccore-extraction-map.png index 51f8cbf..55cd6ae 100644 Binary files a/docs/diagrams/civiccore-extraction-map.png and b/docs/diagrams/civiccore-extraction-map.png differ diff --git a/docs/diagrams/civiccore-extraction-map.svg b/docs/diagrams/civiccore-extraction-map.svg index d4ba924..a5c10c9 100644 --- a/docs/diagrams/civiccore-extraction-map.svg +++ b/docs/diagrams/civiccore-extraction-map.svg @@ -1,125 +1 @@ - -CivicCore v0.11.0 shipped and partial/future namespaces -Architecture map showing CivicCore v0.11.0 shipped primitives on the left and partial or future extraction targets on the right. - - - - - -CivicCore v0.11.0 - Shared platform primitives for CivicSuite modules. Library only; not an end-user app. - - - Shipped implementation - Safe for downstream modules to import in the current development line. - - - - migrations - runner + shared baseline - - - - db - shared SQLAlchemy Base - - - - llm - providers + templates - - - - audit - hash-chained events - - - - provenance - source + citation metadata - - - - connectors - offline manifests + import helpers - - - - exports - bundles + checksums - - - - city_profile - local deployment config - - - - Partial or future extraction targets - Some namespaces ship helper slices; deeper platform behavior remains future work. - - - - auth - shipped bearer helpers - - - - catalog - catalog primitives - - - - exemptions - legal exemption engine - - - - ingest - document ingestion - - - - notifications - shipped notice helpers - - - - onboarding - web onboarding flows - - - - scaffold - module templates - - - - search - shipped helper surface - - - - verification - shipped QA helpers - - - - live connectors - sync + write-back - - -Boundary: v0.11.0 ships helper surfaces for auth, notice compliance, search access/ranking, verification, onboarding, and local imports. It does not ship live sync, ingestion, search indexing, delivery queues, or legal determinations. - +

Planned extraction targets — placeholder __init__.py only, no implementation

civiccore.auth

civiccore.audit

civiccore.catalog

civiccore.connectors

civiccore.exemptions

civiccore.ingest

civiccore.notifications

civiccore.onboarding

civiccore.scaffold

civiccore.search

civiccore.verification

civiccore v1.2.0 - shipped core surface

civiccore.migrations
runner + idempotent guards
civiccore_0001_baseline_v1
civiccore_0002_llm
civiccore_0003_local_task_queue

civiccore.db.Base
shared SQLAlchemy
declarative base

civiccore.llm
providers + templates +
registry + context +
structured output

 \ No newline at end of file diff --git a/docs/diagrams/migration-order.mmd b/docs/diagrams/migration-order.mmd index 1a2fbe3..8f6b951 100644 --- a/docs/diagrams/migration-order.mmd +++ b/docs/diagrams/migration-order.mmd @@ -7,7 +7,8 @@ sequenceDiagram Consumer->>Civic: upgrade_to_head() [subprocess] Civic->>Civic: civiccore_0001_baseline_v1 Civic->>Civic: civiccore_0002_llm (ALTER prompt_templates) - Civic->>CivicVer: stamp 'civiccore_0002_llm' + Civic->>Civic: civiccore_0003_local_task_queue + Civic->>CivicVer: stamp 'civiccore_0003_local_task_queue' Civic-->>Consumer: done Consumer->>Consumer: run consumer-side alembic chain Consumer->>ConsumerVer: stamp '020_phase2_consumer_app_backfill' diff --git a/docs/diagrams/migration-order.png b/docs/diagrams/migration-order.png index 6dfb8ce..7b937c7 100644 Binary files a/docs/diagrams/migration-order.png and b/docs/diagrams/migration-order.png differ diff --git a/docs/diagrams/migration-order.svg b/docs/diagrams/migration-order.svg index 2c26aee..b0be701 100644 --- a/docs/diagrams/migration-order.svg +++ b/docs/diagrams/migration-order.svg @@ -1 +1 @@ -alembic_versionalembic_version_civiccoreciviccore.migrations.runner.upgrade_to_headConsumer app(records-ai env.py)alembic_versionalembic_version_civiccoreciviccore.migrations.runner.upgrade_to_headConsumer app(records-ai env.py)Two separate version tablesprevent revision-name collisionsbetween civiccore and consumers.upgrade_to_head() [subprocess]1civiccore_0001_baseline_v12civiccore_0002_llm (ALTER prompt_templates)3stamp 'civiccore_0002_llm'4done5run consumer-side alembic chain6stamp '020_phase2_consumer_app_backfill'7 \ No newline at end of file +alembic_versionalembic_version_civiccoreciviccore.migrations.runner.upgrade_to_headConsumer app(records-ai env.py)alembic_versionalembic_version_civiccoreciviccore.migrations.runner.upgrade_to_headConsumer app(records-ai env.py)Two separate version tablesprevent revision-name collisionsbetween civiccore and consumers.upgrade_to_head() [subprocess]1civiccore_0001_baseline_v12civiccore_0002_llm (ALTER prompt_templates)3civiccore_0003_local_task_queue4stamp 'civiccore_0003_local_task_queue'5done6run consumer-side alembic chain7stamp '020_phase2_consumer_app_backfill'8 \ No newline at end of file diff --git a/docs/index.html b/docs/index.html index 539d577..0f5cdd6 100644 --- a/docs/index.html +++ b/docs/index.html @@ -131,7 +131,7 @@

What it is

Shipped core surface