feat(docs): rebuild documentation with the Paper design system

Author: emir-karabeg

.github/workflows/docs-negotiation.yml

@@ -29,11 +29,29 @@ jobs:
   http:
     name: docs HTTP
     runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-8vcpu-ubuntu-2404' || 'linux-x64-8-core' }}
-    timeout-minutes: 15
+    timeout-minutes: 20
+    services:
+      search-postgres:
+        image: pgvector/pgvector:pg17
+        env:
+          POSTGRES_USER: docs_test
+          POSTGRES_PASSWORD: docs_test
+          POSTGRES_DB: docs_search_test
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd "pg_isready -U docs_test -d docs_search_test"
+          --health-interval 5s
+          --health-timeout 5s
+          --health-retries 10
     env:
       DATABASE_URL: postgresql://docs_test:docs_test@127.0.0.1:1/docs_test
       DOCS_E2E_BASE_URL: http://127.0.0.1:3001
       DOCS_NEGOTIATION_REPORT_PATH: ${{ github.workspace }}/test-results/docs-negotiation.json
+      DOCS_SEARCH_E2E_DATABASE_URL: postgresql://docs_test:docs_test@127.0.0.1:5432/docs_search_test
+      DOCS_SEARCH_E2E_REPORT_PATH: ${{ github.workspace }}/test-results/search-postgres/report.json
+      DOCS_INTERFACES_REPORT_PATH: ${{ github.workspace }}/test-results/docs-interfaces.json
+      DOCS_CORPUS_REPORT_PATH: ${{ github.workspace }}/test-results/docs-corpus.json
       DOCS_HEADER_REPORT_PATH: ${{ github.workspace }}/test-results/docs-headers.json
       DOCS_NAVIGATION_REPORT_PATH: ${{ github.workspace }}/test-results/navigation/docs.json
     steps:
@@ -47,6 +65,8 @@ jobs:
         run: bun run --cwd apps/docs build
       - name: Verify native header edge cases
         run: bun run --cwd apps/docs test:headers:e2e
+      - name: Verify isolated PostgreSQL search and index lifecycle
+        run: bun run --cwd apps/docs test:search:postgres:e2e
       - name: Install browser for navigation checks
         run: bunx --no-install playwright install --with-deps chromium
       - name: Verify HTTP content negotiation
@@ -56,6 +76,8 @@ jobs:
           DOCS_SERVER_PID=$!
           trap 'kill "$DOCS_SERVER_PID" 2>/dev/null || true' EXIT
           bun run --cwd apps/docs test:negotiation:e2e
+          bun run --cwd apps/docs scripts/test-docs-interfaces-e2e.ts
+          bun run --cwd apps/docs test:corpus:e2e
           bun run --cwd apps/docs test:navigation:e2e
       - name: Upload HTTP regression report
         if: always()

.github/workflows/docs.yml

@@ -30,11 +30,21 @@ jobs:
         uses: ./.github/actions/setup
         with:
           provider: ${{ vars.CI_PROVIDER }}
-          node-version: '22'
+          node-version: '24'
 
       - name: Process docs embeddings
-        working-directory: ./apps/sim
+        working-directory: ./apps/docs
         env:
           DATABASE_URL: ${{ secrets.DATABASE_URL }}
           OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
-        run: bun run scripts/process-docs.ts --clear 
+        run: |
+          bun run content:generate
+          bun run scripts/index-search.ts
+          bun run scripts/index-search.ts --check
+
+      - name: Upload search index report
+        if: always()
+        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
+        with:
+          name: docs-search-index
+          path: apps/docs/test-results/docs-search-index.json

apps/docs/.gitignore

@@ -35,13 +35,10 @@ bun-debug.log*
 *.tsbuildinfo
 next-env.d.ts
 
-# Fumadocs
-/.source/ 
+# Build-generated content
+/.content/
 .plans/
 
-# fumadocs generates .source dirs anywhere a source.config sits
-**/.source/
-
 # Build-generated documentation asset
 /public/llms-full.txt
 /public/.llms-full-*/

apps/docs/ARCHITECTURE.md

@@ -0,0 +1,66 @@
+# Sim docs architecture and authoring
+
+Sim owns the content catalog, page rendering, navigation, search, and machine interfaces. Next.js prerenders public pages from that catalog. MDX, Shiki, Radix primitives, and the official MCP SDK provide focused parsing, highlighting, accessibility, and protocol implementations. No Fumadocs runtime or build dependency remains.
+
+## Directory ownership
+
+| Location | Responsibility |
+| --- | --- |
+| `content/docs/` | Authored MDX, navigation metadata, and generated content regions |
+| `scripts/compile-content.ts` | Build orchestration, validation, cache, and migration report |
+| `lib/content/` | Typed catalog, MDX compilation, Markdown serialization, navigation, and OpenAPI projection |
+| `.content/` | Ignored generated catalog and individually loadable compiled page modules |
+| `components/docs/` | Shared design tokens, shell, navigation, page framing, search, and API reference |
+| `components/ui/` | Shared reading components used by MDX and generated references |
+| `components/workflow-preview/` | Interactive examples and their existing graph/configuration/output data |
+| `lib/search/` | Common search implementation for the UI, HTTP API, and MCP |
+| `lib/http/`, `app/api/` | Bounded public discovery, page retrieval, and MCP adapters |
+| `app/[[...slug]]/page.tsx` | Thin catalog-backed page, metadata, and static-parameter adapter |
+| `app/design-system/` | Local component gallery, excluded from indexing |
+| `migration/` | Baseline inventory, Paper measurements, parity matrix, and migration results |
+
+`app/` follows the App Router's file conventions. Ordinary content does not create a React route or duplicate a layout. The single catch-all route imports a shared article and shell, then loads that page's server-compiled MDX or specification reference. Promised route parameters and the Metadata API follow the installed Next.js 16 documentation.
+
+## Add or change a page
+
+1. Read `EDITORIAL_GUIDE.md` and the complete existing page before editing.
+2. Add `content/docs/<section>/<page>.mdx` with a nonempty `title` and a useful `description`. Folder `index.mdx` files fold to the folder URL. Keep existing paths and explicit `[#anchor]` aliases stable.
+3. Place the page in the appropriate `meta.json` order. Its optional `labels` map overrides a navigation entry's label without changing the article title. The catalog derives navigation, section-local previous/next links, discovery, and the sitemap. Do not add another page list in a consumer.
+4. Use ordinary Markdown and shared components. MDX imports must be named imports registered in `lib/content/compile.ts`. Unknown modules, components, metadata, duplicate routes/anchors, broken internal links, or missing local media fail compilation with source context.
+5. Run `bun run --cwd apps/docs content:generate`, inspect the affected page and Markdown, and run the applicable repository gates.
+
+The compiler records source hashes, heading IDs, code-fence hashes and media references. Run `content:verify-migration` to enforce preservation against the frozen replacement baseline; ordinary authoring permits intentional prose and code updates. It serializes every tab and disclosure into meaningful Markdown, including workflow configuration and outputs. The introductory paragraph is hidden in HTML only when it exactly repeats the page description; machine-readable prose remains intact.
+
+## Extend the component library
+
+Paper is the visual source of truth. Exact board references, dimensions, tokens and intentional differences from platform components are in `migration/paper-design.md`.
+
+Keep reusable chrome in `components/docs/` and reusable reading components in `components/ui/`. Export Markdown element mappings from `getMDXComponents()` rather than styling each page. A new MDX component also needs an explicit compiler allowlist entry and a meaningful serializer in `lib/content/markdown.ts`. Add its states to `/design-system` and inspect its keyboard, touch, light and dark behavior.
+
+Use canonical `@sim/emcn/icons` glyphs. Docs tokens are scoped under `.docs-root`/`.docs-portal`; platform and landing styles are unchanged. The docs `cn` wrapper extends class merging for the Paper text-size tokens, so a color utility cannot erase a typography utility. Shared controls own hover, focus, pressed, disabled, loading, success, and error states.
+
+Default components to server rendering. Client islands are reserved for navigation, search, tabs, disclosure reveal, clipboard, media viewing, theme choice, and interactive workflow examples. Pass rendered server content into those islands. Never import the catalog, compiler, specifications, embedding credentials, or syntax highlighter into a client component.
+
+## Assets and generated references
+
+Keep platform screenshots and original integration logos in `public/static/` with meaningful alternative text. Use the shared `Image` component for sizing, captions and expansion. Preserve workflow fixtures as data; the shared preview renders the actual graph, inspector, and outputs.
+
+Product contracts remain authoritative for OpenAPI. Run `bun run generate:openapi` after changing their owners; the docs compiler projects operations from all registered specifications. CLI, MCP operations, integration content, and marked Agent streaming regions retain their existing generators listed in `EDITORIAL_GUIDE.md`. Do not hand-edit generated regions. Request examples are inert and never execute an API call or accept credentials.
+
+The agent docs manifest continues to publish existing `.mdx` paths and its deliberate corpus exclusions. Run `bun run check:docs-manifest`, `bun run check:agent-stream-docs`, `bun run check:cli-docs`, `bun run check:openapi`, and `bun run check:docs` after relevant changes.
+
+## Delivery, discovery, and performance
+
+One catalog owns HTML, canonical metadata, heading citations, per-page Markdown, JSON, MCP, and sitemap identity. The aggregate `llms-full.txt` deliberately excludes generated operation bodies; individual operation Markdown is complete. See `MACHINE_ACCESS.md` for public contracts, search freshness and real-client verification.
+
+Reading routes use `generateStaticParams` and reject unknown slugs. Compilation and highlighting happen before or during the production build, not in the browser. The compiler caches unchanged MDX and emits separate modules per page; it bounds aggregate catalog size. The shell receives only navigation and headings, while article content remains server rendered. Fonts are local Geist/Geist Mono assets; only the reading font is preloaded. Code and tables scroll within their own surfaces. Reduced-motion preferences suppress decorative transitions.
+
+Metadata uses canonical URLs, descriptive titles, Open Graph/Twitter images, real Git modification dates when available, `TechArticle`/`BreadcrumbList` structured data, and a generated sitemap. JSON-LD is safely serialized. Robots permit public docs and machine-readable discovery. There is no special AI-ranking markup or indexing guarantee: clear HTML, preserved citations, useful text, crawlable links, and consistent public representations are the supported foundations.
+
+The implementation was checked against the installed Next.js 16.4.1 docs and current official [project structure](https://nextjs.org/docs/app/getting-started/project-structure), [server/client boundaries](https://nextjs.org/docs/app/getting-started/server-and-client-components), [MDX](https://nextjs.org/docs/app/guides/mdx), and [metadata](https://nextjs.org/docs/app/getting-started/metadata-and-og-images) guidance. The monorepo's patched Next version is retained. Google's [AI search guidance](https://developers.google.com/search/docs/appearance/ai-features) remains the basis for crawlability; no separate AI-specific schema is required.
+
+## Review and rollback
+
+Review the migration report first, then the library gallery and representative pages, then the HTTP/browser reports. Run a production build without embedding credentials to verify the local fallback. Production search index publication remains a separate deployment step and must pass its freshness check.
+
+This change preserves the docs workspace, origin, deployment identity, product generators, and content paths. Reverting the docs replacement commit and its lockfile/workflow changes restores the prior implementation; it requires no database migration or product-data rollback. Keep the PR in draft until the visual and behavior evidence has been reviewed.

apps/docs/MACHINE_ACCESS.md

@@ -0,0 +1,96 @@
+# Reading Sim documentation from tools
+
+Every interface uses the same compiled catalog as the rendered pages. Documentation access is public and read-only. It does not authorize access to Sim workspaces or execute product API requests.
+
+## Markdown and discovery
+
+Append `.md` or `.mdx` to a canonical page URL, or request that URL with `Accept: text/markdown`:
+
+```sh
+curl https://docs.sim.ai/integrations/slack.md
+curl -H 'Accept: text/markdown' https://docs.sim.ai/integrations/slack
+```
+
+The compatibility path `/llms.mdx/integrations/slack` returns the same complete Markdown. HTTP negotiation respects quality values, defaults to HTML when representations have equal quality, and preserves `Vary: Accept`. Asset and API routes do not negotiate document Markdown.
+
+`/llms.txt` lists canonical pages, including API operations. `/llms-full.txt` contains authored guides and integration references; it deliberately excludes generated API operation bodies. Each API operation has its own complete Markdown representation. `/openapi.json` downloads the combined canonical OpenAPI document.
+
+## JSON API
+
+`GET /api/docs/pages` returns `{ pages, nextCursor, total }`. Each metadata record includes `id`, `path`, canonical `url`, title, description, section, content hash and last-modified time when available.
+
+- `section`: optional `docs`, `api`, `cli`, or `mcp`.
+- `limit`: 1–100, default 25.
+- `cursor`: the previous response's `nextCursor`. Omit it for the first page.
+
+Continue until `nextCursor` is `null`. Unknown cursors and invalid parameters return 400.
+
+`GET /api/docs/page?path=/introduction` returns `{ page }`, containing the same metadata, headings and complete processed `markdown`. Use a canonical site-relative path. Remote URLs, traversal paths and malformed paths return 400; an unknown canonical page returns 404. Content is never silently truncated.
+
+`GET /api/search?query=slack&limit=10` returns the established array of `{ id, type: "page", url, content, breadcrumbs }`. `q` is a supported alias for `query`. Queries are bounded to 256 code points; the result limit defaults to 10 and is capped at 20. Heading citations preserve explicit document anchors. An empty query returns `[]`; an actual service failure returns 503.
+
+## Documentation MCP
+
+Configure a remote Streamable HTTP server at `https://docs.sim.ai/api/mcp`. This endpoint is separate from Sim's workspace MCP server and `/mcp/...` documentation pages. It requires no API key.
+
+The official stable MCP TypeScript SDK 2.3.1 serves the 2026-07-28 protocol and stateless compatibility for 2025-era clients. Tools:
+
+| Tool | Input | Result |
+| --- | --- | --- |
+| `list_pages` | Optional `section`, `cursor`, `limit` as above | Paginated metadata and next cursor |
+| `get_page` | `path` | Complete Markdown and canonical metadata |
+| `search` | `query`, optional `limit` 1–20 | The same search results as the docs UI |
+
+All three tools advertise read-only behavior. Unknown pages produce tool errors; invalid inputs produce protocol/tool validation errors. The transport rejects unapproved browser origins, unrecognized hosts and request bodies over 16 KiB. Non-browser MCP clients normally omit `Origin`. No caller-supplied URL is fetched.
+
+## Search index operations
+
+Basic search uses a server-only index generated atomically with the local compiled catalog and needs no database or embedding credentials. Markdown parsing and text normalization happen during content compilation; requests read precomputed chunks. The generated index is capped at 64 MiB and its catalog fingerprint is validated before serving results. When `DATABASE_URL` is present, bounded PostgreSQL keyword search can supplement it; `OPENAI_API_KEY` additionally enables vector search using the existing 1536-dimensional `text-embedding-3-small` table contract. Unavailable providers fall back to lexical results. Results with old content hashes or nonexistent heading targets are excluded.
+
+From `apps/docs`:
+
+```sh
+bun run content:generate
+bun scripts/index-search.ts --plan
+bun scripts/index-search.ts --check
+bun scripts/index-search.ts
+```
+
+`--plan` is local and read-only. `--check` requires the search database and reports missing, stale or retired documents without writes. The final command publishes changed pages with database and embedding credentials: it prepares bounded embedding batches, atomically replaces one document at a time, and removes retired documents only after all page updates succeed. A failed page leaves its previous index intact. The canonical index includes authored content and generated API operations; the agent VFS retains its deliberate `academy` and `api-reference` exclusions.
+
+Set `DOCS_SEARCH_REPORT_PATH` to choose the JSON report destination. The default is `test-results/docs-search-index.json`. The main-branch docs workflow runs publication and freshness checks; no production indexing is part of the docs build.
+
+## Repeatable verification
+
+Start a local production server, then run the existing negotiation and navigation suites and the official-client interface suite:
+
+```sh
+DOCS_E2E_BASE_URL=http://127.0.0.1:3001 \
+DOCS_INTERFACES_REPORT_PATH=test-results/interfaces.json \
+bun scripts/test-docs-interfaces-e2e.ts
+```
+
+The suite exercises real HTTP and the official MCP client, including discovery pagination, complete page retrieval, search consistency, invalid inputs, denied origins and byte limits. `DOCS_NEGOTIATION_REPORT_PATH` and `DOCS_NAVIGATION_REPORT_PATH` configure the companion reports. CI uploads the reports and browser screenshots on success or failure.
+
+
+The PostgreSQL suite uses a new, empty, loopback database and no embedding key. It verifies real keyword search, citation freshness, graceful database failure, repeated index publication, retirement and failed-publication preservation. It refuses databases containing existing tables. CI provisions its own `pgvector/pgvector:pg17` service and uploads the report with the other docs checks.
+
+To repeat locally, create a disposable container (the command returns its ID):
+
+```sh
+docker run --rm -d --name sim-docs-search-e2e \
+  -e POSTGRES_USER=docs_test -e POSTGRES_PASSWORD=docs_test \
+  -e POSTGRES_DB=docs_search_test -p 127.0.0.1:15432:5432 \
+  pgvector/pgvector:pg17
+docker exec sim-docs-search-e2e pg_isready -U docs_test -d docs_search_test
+```
+
+After PostgreSQL reports ready, run from `apps/docs`:
+
+```sh
+DOCS_SEARCH_E2E_DATABASE_URL=postgresql://docs_test:docs_test@127.0.0.1:15432/docs_search_test \
+DOCS_SEARCH_E2E_REPORT_PATH=test-results/search-postgres/report.json \
+bun --no-env-file --conditions=react-server scripts/test-search-postgres-e2e.ts
+```
+
+Then stop the container using the ID returned by `docker run`; `--rm` removes its temporary data. Successful vector generation and provider ranking still require separate verification with an embedding provider. The local suite makes no paid provider calls.