Phase 1-2: foundation, data model and tenancy - #1
Merged
Merged
Conversation
Scaffolds the Next.js/TypeScript project and the multi-tenant core: - Validated environment contract (src/server/config/env.ts) — the only place process.env is read. - PostgreSQL + pgvector schema: organizations, users, memberships, sessions, integrations, documents, chunks, sync_jobs, query_logs. Every tenant-owned row carries organization_id; chunks get an HNSW cosine index for retrieval. - Session auth: scrypt password hashing, HMAC-signed opaque session cookies, membership re-checked on every request. - Cross-cutting infrastructure: error taxonomy, structured JSON logger, route wrapper (request id/timing/error translation in one place), token-bucket rate limiter, jittered retry helper, Prometheus metrics. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6VZimco9iG6Upxf23mke6
Provider-agnostic pipeline between an integration and pgvector: - NormalizedDocument is the contract every integration produces; nothing downstream knows a provider name. - Embedding provider abstraction with Voyage, OpenAI and a deterministic local implementation (a signed hashing vectorizer) so dev, CI and the test suite run the full path offline with no API key. - Structure-aware chunking: packs whole paragraphs to a token target, breaks at markdown headings, tracks a depth-indexed heading trail, and carries sentence-aligned overlap across boundaries. - Idempotent ingestion: a document hash short-circuits unchanged content, and per-chunk hashes let an edit to one paragraph re-embed only that paragraph. Deletions soft-delete the document and drop its chunks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6VZimco9iG6Upxf23mke6
Establishes the SourceIntegration contract and implements all three connectors against it. Adding a source touches its own directory plus one line in the registry. - SourceIntegration: OAuth (authorize/exchange) plus an async generator yielding pages of NormalizedDocuments with a cursor, so a sync is resumable page by page rather than all-or-nothing. - Notion: OAuth, block tree rendered to Markdown (headings preserved for the chunker), incremental sync off a last_edited watermark. - Google Drive: native-format export, transparent access-token refresh on 401, incremental sync off the changes feed. - Slack: threads as documents (a lone message is not knowledge), mrkdwn rendered to readable text, per-channel watermarks. - OAuth credentials encrypted at rest with AES-256-GCM; signed OAuth state carries the tenant through the redirect. - Postgres job queue using FOR UPDATE SKIP LOCKED, idempotency keys that collapse duplicate enqueues, jittered retry backoff, dead-lettering and a reaper for jobs abandoned by a crashed worker. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6VZimco9iG6Upxf23mke6
- searchChunks fuses pgvector cosine search with Postgres full-text search via Reciprocal Rank Fusion. Vector search alone misses exact tokens (error codes, policy numbers); keyword search alone misses paraphrase. Both arms filter on organization_id in their own WHERE clause, so another tenant's rows are never candidates. - Per-document cap keeps one long page from filling the context window. The cap is hard — fewer results rather than backfilling from the document it just limited. - Answer generation with Claude: frozen cached system prompt, structured output for citations so source numbers are validated against what was actually retrieved (invented indices are dropped), refusal stop_reason handled, per-status error chain. Degrades to retrieval-only without a key. - Every query is logged with its retrieved chunks, successes and failures alike, so answer quality is reviewable after the fact. Tests: full ingestion path and retrieval against real Postgres + pgvector, including two tenants holding contradictory answers to the same question. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6VZimco9iG6Upxf23mke6
Web app (Next.js App Router): - Ask, Dashboard, Sources and Documents pages. Server components read the session and query scoped to its organization; client components own only transient request state — no business logic in React. - Session auth pages, and a route wrapper that turns AppError into HTTP in one place so route handlers stay a few lines each. - Hand-written design system (tokens, primitives, components) with light and dark palettes; no CSS framework for a surface this small. Slack interface: - /contextbridge slash command with HMAC signature verification and a replay window. Slack allows three seconds, so the handler acknowledges immediately and posts the cited answer to response_url out of band. - The Slack team id maps an inbound command to its tenant. Also: health check that actually exercises Postgres and pgvector, Prometheus metrics endpoint, and a seed script that runs demo content for three sources through the real ingestion pipeline. OAuth callbacks moved to /api/oauth/[provider]/callback: they collided with /api/integrations/[id]/* on Next's dynamic segment naming rule. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6VZimco9iG6Upxf23mke6
73 tests, all against real Postgres + pgvector. Two real bugs found and fixed: - Raw SQL returns snake_case columns, so every multi-word field on a claimed job (integrationId, organizationId, maxAttempts...) read as undefined at runtime while still type-checking. That broke the worker outright. Added an explicit row type and mapper at the raw-SQL boundary. - A credential decryption failure was classified retryable, so the worker burned all five attempts on a job that can never succeed. It now raises a non-retryable error naming the likely cause (CREDENTIAL_SECRET rotated after the source was connected) and dead-letters on the first attempt. Coverage: queue idempotency and revival, FOR UPDATE SKIP LOCKED handing a job to exactly one of several racing workers, backoff vs. dead-lettering, reclaiming a crashed worker's jobs; sync cursor checkpointed per page so a mid-sync crash resumes, full vs. incremental deletion semantics; password hashing, credential encryption tampering, session cookie forgery, OAuth state re-pointing, Slack signature and replay window; membership revocation taking effect immediately. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6VZimco9iG6Upxf23mke6
- ADRs 001-004 as required by DECISION.md: pgvector as the single datastore, modular monolith, asynchronous ingestion, and the organization-level multi-tenancy strategy. Each records the alternatives considered and the consequences accepted, including what was deliberately deferred (Postgres RLS, shared rate limiters, webhook receivers). - README covering the quick start, how ingestion/chunking/retrieval/ citation actually work, how to add a source, and an explicit list of production gaps rather than implying there are none. - CI: typecheck, migrate, test and build against a real pgvector service container. No API keys needed — the local embedding provider runs the full path offline. - scripts/load-env.ts: Next.js loads .env itself but a plain tsx process does not, so `npm run worker` came up with no DATABASE_URL while `npm run dev` worked. The documented quick start now works as written. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6VZimco9iG6Upxf23mke6
Three cleanups found reviewing the finished code: - Google Drive refreshed expired access tokens but nothing persisted them: persistRefreshedCredentials was exported and never called, and the client's didRefresh flag was never read, so every sync paid for a fresh refresh round trip. Rotated credentials now come back through an onCredentialsRefreshed callback on SyncContext — a callback rather than a direct write so connectors stay unaware of the database and the driver keeps ownership of encryption. - Dropped the document.ingest and document.delete job types. Nothing enqueued them and the worker handled them by falling through to a full sync, which is not what their names promise. - Split the Slack Block Kit formatter out of the module named "verify" into slack/format.ts, leaving slack/signature.ts to do one thing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F6VZimco9iG6Upxf23mke6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scaffolds the Next.js/TypeScript project and the multi-tenant core:
place process.env is read.
sessions, integrations, documents, chunks, sync_jobs, query_logs.
Every tenant-owned row carries organization_id; chunks get an HNSW
cosine index for retrieval.
cookies, membership re-checked on every request.
route wrapper (request id/timing/error translation in one place),
token-bucket rate limiter, jittered retry helper, Prometheus metrics.