fix(compliance-checks): make serverless compliance job runnable (deploy backend source + __file__-free bootstrap)

[surojitchowdhury]

Fixes #685

Summary (plain language)

Ontos runs its compliance checks as a scheduled Databricks job on serverless compute. That job could not run on serverless at all, for two compounding reasons:

1. NameError at module load — the entry script's first executable line derived its own folder from Python's __file__. On serverless the script is executed via exec(compile(...)) in a namespace where __file__ is not bound, so Path(__file__) raised NameError before main() ran.
2. ModuleNotFoundError: No module named 'src' — even past that, the job does from src.controller.compliance_manager import ComplianceManager at runtime, but the backend src package was never deployed to serverless (the deployer only uploaded the workflow folder). So the run connected to the database, loaded policies, then crashed on the import.

This PR makes the job genuinely runnable on serverless: the backend src package is packaged and shipped alongside the workflow, extracted locally at runtime, and put on sys.path; and the module-load bootstrap no longer depends on __file__.

Root cause confirmed on the real serverless job

The production Compliance Checks job on serverless fails today at exactly this point (real run, current code):

✓ Database connection established
Found 26 policies to run
Initializing ComplianceManager...
→ from src.controller.compliance_manager import ComplianceManager
ModuleNotFoundError: No module named 'src'

Change (5 files)

• utils/workspace_deployer.py — when a workflow sets deploy_backend_source, build a backend_src.zip of the backend src package and upload it beside the deployed workflow script.
• controller/jobs_manager.py — when deploy_backend_source is set, pass the uploaded archive's /Workspace path to the job as --backend_source_path (with a directory fallback when no archive is deployed).
• workflows/compliance_checks/compliance_checks.py
  • __file__-free module bootstrap: with __file__ present, behaviour is unchanged; without it, the module-level insert is guarded so import never aborts.
  • Off-by-one fix: the backend source root is src/backend (not src/backend/src), so from src.* resolves.
  • New _add_backend_source_path(): if --backend_source_path is a .zip, extract it to a local temp directory (validated to contain the src package, with a zip-slip / path-traversal guard on every member) and prepend that local directory to sys.path; if it is a directory, insert it directly. This is required because CPython cannot zipimport from the /Workspace FUSE mount — a naive "put the zip on sys.path" approach raises NotADirectoryError on serverless.
• workflows/compliance_checks/compliance_checks.yaml — enable deploy_backend_source: true, add the --backend_source_path job parameter, and expand the serverless environment dependencies to cover the import closure (gitpython, pydantic[email], pydantic-settings, …).
• tests/test_compliance_checks_workflow.py — hermetic coverage that builds the real archive, strips the backend root from sys.path, execs the module without __file__, and asserts both lazy imports (src.controller.compliance_manager, src.db_models.compliance) resolve from the extracted local directory — not the zip or the repo. Plus both-branch fallback and normal-__file__ cases.

Serverless evidence — the fix works on real compute

Deploying the fixed script + backend_src.zip and running on serverless, the exact import that ModuleNotFoundErrors in production now succeeds:

Backend source added to sys.path: /tmp/ontos-compliance-backend-XXXX
  (local extraction of .../backend_src.zip)
✓ Database connection established
Found N policies to run
Initializing ComplianceManager...
  ✓ ComplianceManager initialized          ← the import that fails in prod now succeeds
[POLICY] ...  (proceeds to evaluate policies)

Gate output

$ hatch -e dev run pytest backend/tests/test_compliance_checks_workflow.py \
    backend/tests/test_compliance_dsl.py -q
======================= 45 passed =======================
$ hatch -e dev run ruff check .../compliance_checks/compliance_checks.py
(no new violations on changed lines)

Nothing under .github/workflows/ is touched.

Full end-to-end serverless validation

The fix was validated with a real serverless run of the deployed Compliance Checks job against a live Ontos Lakebase (ontos-db, schema app_ontos, 9 active policies, 13 data products):

Backend source added to sys.path: /tmp/ontos-compliance-backend-... (local extraction of backend_src.zip)
✓ Database connection established
Found 9 policies to run
✓ ComplianceManager initialized

The job runs to completion — the from src.* imports that previously raised ModuleNotFoundError now resolve on serverless. This confirms the #685 fix end-to-end.

Scope & follow-up (#807)

This PR is scope-limited to the original issue #685 (the serverless import / __file__ startup failure). The end-to-end run surfaced a separate, deeper bug in the compliance workflow — filed as #807 — which is out of scope for this PR and will be fixed there:

The standalone job never calls init_config(), so ComplianceManager.run_policy_inline → get_workspace_client() → get_settings() raises RuntimeError: Settings not initialized. The job completes but evaluates 0 entities (all policies score 0%). Distinct from #685 (which only concerns import/__file__ on serverless).

---

Re-raised from a branch on this repository (previously #725, opened from a fork) so the required CI workflows can run — fork PRs do not receive the internal JFrog OIDC token, which failed every check at setup before any test ran. #725 is closed in favour of this PR.

[mvkonchits-db]

Reviewed with the code-review skill. The fix correctly removes the import-time NameError from referencing __file__ at module load, but I don't think it makes the workflow actually runnable on serverless — leaving as a comment rather than approving so it can be verified.

1. The crash may just move later. On serverless the bootstrap now inserts nothing on sys.path. Module load then succeeds, but main() later hits lazy imports like from src.controller.compliance_manager import ComplianceManager and from src.db_models.compliance import CompliancePolicyDb. The serverless env spec installs only databricks-sdk/sqlalchemy/psycopg2-binary (not the ontos backend), so unless the runtime already has the backend src parent on PYTHONPATH, those raise ModuleNotFoundError — the same job failure, just later.

2. Possible off-by-one in the non-serverless path. The insert uses Path(__file__).parent.parent.parent = src/backend/src (the src package dir itself). But from src.* resolves only when src/backend (the package's parent) is on sys.path. Inserting the package dir makes import src unresolvable in any env that relies solely on this insert.

3. Tests don't cover the failure point. Both new tests set __name__ != '__main__', so main() never runs and the lazy from src.* imports are never exercised — the suite passes without touching the thing that actually fails on serverless. Also assert sys.path == sys_path_before is order-dependent and can flake if a real dep import mutates sys.path first.

Suggestion: validate against a real serverless run (or a test that invokes main()/the lazy imports under a simulated serverless sys.path) before merging, and double-check the insert targets src/backend, not src/backend/src.

[surojitchowdhury]

Thanks for the careful review, @mvkonchits-db — all three points are addressed in the reworked branch, and I validated it with a real end-to-end serverless run.

1. "Crash just moves later" / backend not on serverless — fixed by actually deploying the backend. The deployer now builds a backend_src.zip of the backend src package and uploads it beside the workflow (WorkspaceDeployer), the job passes its /Workspace path via --backend_source_path (JobsManager), and the script extracts it to a local temp dir and puts that on sys.path — extract-to-local, because CPython can't zipimport from the /Workspace FUSE mount. The env spec is expanded to cover the import closure (gitpython, pydantic[email], pydantic-settings, …).

Verified on a real serverless run against a live Ontos Lakebase (9 policies, 13 data products): the job connects, Found 9 policies to run, and ✓ ComplianceManager initialized — the from src.controller.compliance_manager import ComplianceManager line that previously raised ModuleNotFoundError now resolves, and the job runs to completion.

2. Off-by-one — fixed. The bootstrap now targets src/backend (…parent.parent.parent.parent), and the deployer's directory fallback resolves to src/backend as well, so import src resolves.

3. Test coverage — the failure point is now exercised. A new hermetic test builds the real archive, strips the backend roots from sys.path, runs the extract path, and asserts both lazy imports (src.controller.compliance_manager, src.db_models.compliance) resolve from the extracted directory (not the zip, not the repo). The order-dependent sys.path == assertion was replaced with targeted present/absent checks.

Scope note: this PR is intentionally scope-limited to #685 (serverless import / __file__). The end-to-end run surfaced a separate, deeper bug — the standalone job never calls init_config(), so policy evaluation later fails with Settings not initialized and evaluates 0 entities — which is filed as #807 and will be fixed there, not here.