Skip to content

Port the backend to Go and refresh README demo previews - #115

Merged
debpalash merged 1871 commits into
mainfrom
go-rewrite
Oct 29, 2026
Merged

debpalash merged 1871 commits into
mainfrom
go-rewrite

Conversation

@velixio

@velixio velixio commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Replace the Python application runtime with native Go API, worker, scheduler, MCP and migration binaries. Python remains in the internal browser service for Playwright and browser-dependent scraping.

The port covers authentication and workspace isolation, settings and encrypted secrets, workbooks and enrichment, leads and signals, audiences and destinations, automations, outreach, governance and agent access. PostgreSQL with forced workspace RLS owns data and the durable queue; Redis carries progress. The Go migration runner adopts the exact Alembic baseline e7b2c4d6f8a0 and applies migrations through version 4.

Final backend fixes

  • Scope person-dependent provider caches to their inputs and settings; do not reuse shared-domain lead scores.
  • Recognize known free native providers while preserving explicitly declared prices.
  • Save multi-field encrypted provider settings atomically, with rollback and encryption-failure regressions.
  • Write destination insert timestamps in UTC, covered under UTC, Asia/Kolkata and America/New_York.
  • Build filesystem statistics helpers on Darwin and Linux.

README and deployment notes

The README now includes light/dark dashboard screenshots and two compressed GIFs showing connected records and an import review. These are explicitly labeled as previews from the separate dashboard redesign; this PR does not bundle that local redesign or canonical-record migration 5.

The server example uses a reviewed Go revision instead of assuming an older release tag contains this runtime. Existing Python installations have a separate upgrade section linked to the cutover runbook.

Release images use explicit opengtm-{api,worker,mcp,migrate,scheduler,browser} names. The plain legacy ghcr.io/debpalash/opengtm alias stays unchanged.

Validation

  • Full CI passed on backend fix commit f6abd818: Go vet, race unit/integration tests with PostgreSQL and Redis, shipping builds, frontend lint/tests/build, and browser-service checks. The integration suite includes the real API/worker scenarios with no Python application runtime.
  • All final-head checks passed on 0d973725: Go vet, race unit/integration tests and shipping builds; frontend lint/tests/build; browser-service lint/tests. No backend code changed in the README/assets commit.
  • Targeted local race regressions and generated-SQL consistency checks passed. GIFs were captured from the rendered local app using fictional records; both are under 3 MiB, and README asset links resolve.
  • Local native cutover/restore rehearsal passed against the versioned e7b2c4d6f8a0 reference schema using tracked Python source b8bfc57: Python login and workbook creation, verified PostgreSQL/data-directory backups, Go adoption and job tenant/executor backfill, legacy-token authentication in Go, native workbook jobs, then backup restoration and Python login/workbook access. The restored queue returned to its pre-cutover Python state. Fixture restore plus Python restart took 1.83s; this is not a production recovery-time estimate.

Merge checks

  • Publish isolated backend fixes and README assets.
  • Document explicit release image naming.
  • Pass the isolated native reference-schema cutover/restore rehearsal.
  • Confirm all checks on the final head.

Production cutover preflight

  • Perform the deployed Docker staging cutover/rollback drill with the installation's actual previous Python release and data. The local reference-schema fixture does not verify upgrading an older Alembic schema or running the full container topology.

Existing Python production cutovers must retain a verified pre-cutover backup and pass the rollback rehearsal before deployment. See migration ownership and preconditions and Python-free cutover. No release tag, image publication or production deployment is part of this PR merge.

…erson writes Python's person graph

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…g or mismatched workspace refused, failure hook without one does nothing) (failing)

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…onciliation, Postgres ledger/store/source_stats adapters and live web, browser, settings and enrichment ports; tenant only from the claimed row

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
… unfinished probe is nil) (failing)

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…t and probes without the Python worker

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
… GO_EXECUTOR_BULK_ENRICH, hand-back otherwise) and the in-process source probe (GO_NATIVE_SOURCE_PROBE)

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…CUTOR_COLLECT cohort (failing)

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…ead-less rows are refused (review wave2a M4)

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…r + validate_lead_light) with leadpeople account fit; gate decisions match the recorded Python ones

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…on queries

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…tion

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…dd ValidateOutputColumns

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
… refuse non-literal output webhook URLs with a 422

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…K cohort; preview served behind GO_SOURCE_PREVIEW_ENABLED (RED)

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…rces/{col_id}/preview in Go behind GO_SOURCE_PREVIEW_ENABLED

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…rage counts it as ported behind its flag

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…utor when the cohort covers the workspace (RED)

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…GO_EXECUTOR_SOURCE_WORKBOOK (default python)

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…d GO_ALL_NATIVE (RED)

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…ugh config.Getenv and InCohort (wildcard cohort, GO_ALL_NATIVE)

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…, events), cancel during enrichment, intent block, failure-hook reconciliation, ledger guards, upsert, source_stats and bulk_enrich

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…or, never without a workspace

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
… the GO_EXECUTOR_COLLECT / GO_EXECUTOR_BULK_ENRICH cohort

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…flags, native probe only with the source health executor

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
velixio and others added 18 commits October 28, 2026 13:27
…ect_phone, personal_email) key per person; manifest providers use the cache (failing)

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…_email are keyed per person; manifest providers use the enrichment cache again

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…ct the member to fail at the search (deviation map, README line)

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
… are rate limited, 429 with Retry-After; no separate invite/accept routes exist in this API

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…d and settled against the workspace daily LLM cap; pre-route, descriptions, memory and briefs make no LLM call in Go

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…s and log texts now describes the in-process runtime

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
… production adapters; recorded fixtures replay (failing)

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…outreach store and send queue (enqueue only, in the write transaction, caller keys namespaced per workspace) and autosvc rule validation; stored rules keep the client's key order

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…ne send in each (failing)

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…ng-send check to the workspace (no cross-tenant existence oracle or blocking); QueueSend takes a pgx.Tx

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…mplementation, runs in the pipeline tx)

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…_email and create_automation in Go

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
… sandbox starts on Ubuntu 24.04 runners

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…runner outlasted the one-minute window: 401 instead of 429)

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…package or sequence insert can take

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…reeRecordIDs free recorded literal ids below it

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
…low testpg.SeqFloor; no test moves a shared sequence backwards (fixes workbook_rows pkey collisions across packages under -p 4)

Signed-off-by: Shimul Debnath <270455167+velixio@users.noreply.github.com>
Signed-off-by: debpalash <4178343+debpalash@users.noreply.github.com>
@velixio
velixio marked this pull request as ready for review October 29, 2026 14:41
@velixio
velixio requested a review from debpalash as a code owner October 29, 2026 14:41
@velixio velixio self-assigned this Oct 29, 2026
Signed-off-by: debpalash <4178343+debpalash@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants