implementation-intelligence-hub

A self-hosted Python platform that turns multi-source implementation evidence — Slack, Jira/Linear, email, calendar, recorded meetings, customer-shared screenshots, kickoff PDFs, Salesforce — into evidence-cited daily briefs, slip-risk predictions, and drafted customer status emails for B2B SaaS Implementation Managers.

This is the marquee project in the AI Copilots & Agents portfolio category. It is intentionally not "another GPT wrapper": it is the production-grade pattern enterprises actually need before they will trust an agent with customer-facing work — **multi-modal ingestion + a custom agent loop with reflection + hybrid retrieval + evidence anchoring + a policy engine

• HITL gating + a continuous evaluation harness with backtest replay + multi-tenant isolation + a full audit trail + cost/latency observability** — all runnable on make demo in under 60 seconds with zero real credentials.

Problem domain

A typical Implementation Manager at a mid-market B2B SaaS company owns 8-15 concurrent customer implementations, each running 30-180 days, each generating evidence across:

• Customer-shared Slack Connect channel (50-300 messages/week)
• Internal Slack channel for the deal (parallel discussion)
• Jira / Linear project (tickets, status changes, comments)
• Recorded weekly customer meetings (45-60 min audio each)
• Email threads with customer stakeholders + internal
• Kickoff deck, SOW, contract terms (PDFs)
• Customer-shared screenshots / GIFs of bugs or questions
• CRM account record (Salesforce / HubSpot)

The IM's job — knowing the true state of each implementation, predicting which will slip, drafting honest weekly status updates — currently consumes 8-12 hours/week of manual synthesis per IM and produces reports that are wrong by Monday morning. Slip detection is gut-feel. Blockers surface weeks after they actually started. Onboarding drives the bulk of early churn risk; this is real money.

Existing onboarding tooling (Rocketlane, OnRamp, GUIDEcx, Arrows, Vitally, Catalyst) is project management UI with bolted-on summarization, not multi-modal reasoning over the evidence. This platform is the latter.

What it does, end to end

sources ──► ingest ──► hybrid retrieval ──► agent loop ──► policy engine ──► console + webhooks
(6 types)   dedupe +    FTS · pgvector ·    retrieve →     first-match      timeline · workbench ·
            embed +     structured → RRF    reason →        wins (YAML)      review queue · eval ·
            persist                         cite →                           audit · ops
                                            reflect →
                                            slip_score

1. Ingest six heterogeneous sources through one adapter protocol. Each record is deduped twice — by source key (DB unique constraint) and by content SHA-256 — then embedded and persisted as one evidence row. Screenshots and PDFs run through OCR; meeting recordings through transcription, before the same normalization path.
2. Retrieve with a hybrid fusion of Postgres full-text search, pgvector cosine similarity, and a recency-ordered structured filter, combined via reciprocal-rank fusion (RRF).
3. Reason with a custom 5-step agent loop — retrieve → reason → cite → reflect → slip_score — that emits one of three brief kinds: daily brief, weekly status, or a drafted customer email.
4. Anchor every claim to the exact evidence rows that support it. The reflect step drops any observation claim it cannot cite.
5. Score slip risk green / yellow / red with explicit reasoning and weighted factors.
6. Route the output through a first-match-wins YAML policy engine (auto-publish, route-to-review, escalate).
7. Gate anything customer-facing through a human-in-the-loop review queue.
8. Audit every state transition with before/after snapshots, and emit webhooks the revenue-ops-automation-hub workflow engine can consume.

The agent loop

No framework — a compact orchestrator (~300 lines) with a pluggable step protocol, so the reasoning is fully inspectable and a framework swap stays trivial. Each step's input and output is captured in the brief's agent trace.

Step	What it does
retrieve	Hybrid retrieval (FTS + pgvector + structured) → RRF fusion over the implementation's evidence in the checkpoint window.
reason	Generates the brief body for the requested kind (daily / weekly / drafted email), referencing evidence by source tag.
cite	Parses each claim and binds it back to evidence rows by source + timestamp; persists claims + citations.
reflect	Critiques the draft; drops uncited observation claims; re-cites against the revised body. Records the diff in the trace.
slip_score	Bands the implementation green / yellow / red via structured output, with reasoning + weighted factor list.

The loop runs against a mock LLM by default (transparently heuristic, clearly labelled) so the full demo needs no API key. Set LLM_PROVIDER=anthropic + ANTHROPIC_API_KEY and the exact same loop runs against Claude.

Multi-modal ingestion

Every source lands through the same adapter protocol and normalizes to one evidence schema, so the agent never branches on modality.

Source	Modality	Handling
Slack	text + threads + reactions + image attachments	Web API + Events webhook
Jira / Linear	tickets, transitions, comments	REST API
Gmail	email threads	Gmail API
Google Calendar	meetings + attendance	Calendar API
Salesforce	account status + stage changes	REST API
Recordings	audio → transcription with speaker labels	Whisper (local) / Zoom API
Screenshots	image → OCR + vision reasoning	Tesseract / Claude vision
PDFs (kickoff, SOW)	document text	PDF text extraction

Local-first by default: transcription, OCR, and embeddings all run without a cloud key for the demo; cloud APIs are opt-in for production accuracy.

Operator console

Open http://localhost:8000 and sign in with the credentials printed by make demo. The console surfaces:

• Per-implementation timeline — the central UI primitive. All evidence across all sources, color-coded, with generated briefs interleaved at their checkpoint times.
• Brief workbench — each claim is hover-coupled to its evidence panel (fully client-side, zero-network for review) so an IM can verify provenance before anything reaches a customer.
• Slip-risk banding — green / yellow / red with reasoning and factor chips.
• HITL review queue — drafted customer emails awaiting approval; every approve/reject is audit-logged.
• Sources inspector — per-tenant OAuth connect / disconnect, masked credentials, live vs. mock mode.
• Policy registry — first-match-wins YAML rules, reviewable by ops without a code change.
• Eval dashboard — backtest metrics with trend lines; drill into any metric's underlying cases.
• Audit log + Ops — every state transition, plus cost/latency observability per agent run.

Evaluation harness

make eval replays the agent across a synthetic dataset (multiple implementations × multiple daily checkpoints, with known slip outcomes) and scores four metrics, each persisted with a delta vs. the chosen baseline:

Metric	Question it answers
Slip calibration (Brier)	Are predicted bands honest probabilities?
Blocker recall × lead time	How early, and how often, is a known blocker flagged?
Citation precision	What fraction of claims are backed by real evidence?
Brief acceptance rate	Would a (synthetic) IM accept the brief unedited?

Ground truth is derived from the synthetic scenarios, not from the agent's own output, so the eval is honest. A regression gate (iih eval --strict) fails CI when any metric regresses beyond its threshold vs. the baseline.

Multi-tenant from day one

• Per-tenant credential keys derived via HKDF-SHA256 over the platform key
  • tenant_id — a ciphertext encrypted for one tenant cannot be decrypted under another's key. Decryption falls back to the platform key for pre-migration rows.
• Row-level isolation via tenant_id on every row.
• ConnectorAccount.set_credentials() round-trips OAuth bundles through the per-tenant cipher; masked_credentials() returns a UI-safe view.
• The AuditLog table captures every state transition with before_state / after_state snapshots so "what changed and when" is always answerable — every HITL approve/reject and policy decision writes one.

Stack

• Python 3.12
• FastAPI · Uvicorn · Jinja2 · HTMX · Alpine.js
• SQLAlchemy 2.0 · Alembic · PostgreSQL 16 with pgvector
• Redis 7 · RQ
• pydantic 2 · pydantic-settings · structlog
• cryptography (Fernet + HKDF) · bcrypt · itsdangerous
• Anthropic SDK · sentence-transformers · Whisper · Tesseract
• httpx · tenacity · PyYAML
• Typer · Rich
• pytest · ruff · black · mypy

Opt-in extras pinned in pyproject.toml:

• iih[retrieval] — sentence-transformers + rank-bm25 + numpy
• iih[llm] — anthropic + openai (mock by default)
• iih[audio] — faster-whisper (mock by default)

Quickstart

1) Bring up the data plane

docker compose up -d postgres redis

Postgres runs on :5434 (image: pgvector/pgvector:pg16). Redis runs on :6381. Ports are deliberately offset from the other portfolio repos so all the AI projects can run simultaneously without collision.

2) Install + migrate + seed

cp .env.example .env
make install
make db-upgrade        # runs migrations (incl. CREATE EXTENSION vector)
make demo              # seeds tenant + admin, ingests synthetic data, runs the agent

.env.example defaults keep CONNECTOR_MODE=mock and every API key as __PLACEHOLDER__. The platform runs end-to-end without any real LLM / SaaS credentials — the live adapters sit behind the same interface and flip on when real OAuth credentials are configured.

3) Explore the console + run the eval

make serve            # operator console on :8000
make eval             # backtest harness over the synthetic dataset

MCP server

The platform exposes itself over the Model Context Protocol so an external assistant — Claude Desktop, an IDE, or another agent — can drive it as a set of tools instead of through the HTTP console. The server runs over stdio:

iih mcp

Six tenant-scoped tools are registered (every call names a tenant by slug, so one tenant can never read another's evidence or briefs):

Tool	What it does
list_implementations	All implementations for a tenant with status + latest slip band
get_implementation	One implementation's detail plus its most recent briefs
search_evidence	Hybrid (lexical + vector + structured) search over an implementation's evidence
generate_brief	Run the agent and persist a brief (daily_brief / weekly_status / drafted_email) — a write tool
get_brief	Fetch a brief with its claims and per-claim citations
latest_slip_assessment	Most recent slip-risk band + reasoning + factors

The tool bodies are thin wrappers in iih/mcp/server.py over the pure, individually-testable functions in iih/mcp/tools.py — read tools open a read scope, the one write tool commits. Wire it into a client (e.g. Claude Desktop's claude_desktop_config.json) as:

{
  "mcpServers": {
    "iih": { "command": "iih", "args": ["mcp"] }
  }
}

Tests

make test       # unit + integration (integration runs against local Postgres)
make lint       # ruff + mypy
make format     # black + ruff --fix

Composing with revenue-ops-automation-hub

This repo emits implementation.brief.published, implementation.slip_alert, and implementation.hitl.opened webhooks into the revenue-ops-automation-hub workflow engine — mirroring the existing composition between vci-hub and RevOps Hub, reinforcing the "portfolio as a coherent platform" story.

License

MIT. See LICENSE.

About this code

Open-source companion to the agentic AI and multi-modal reasoning work done by acilox. For paid implementation, on-prem deployment, or extension of these patterns into a production environment, open an issue.