Repository navigation
fix(firecrawl): unwrap double-data response + Slack DM on approvals - #20
Merged
Merged
Conversation
Pivots the project from a sales/CS/RevOps GTM team to an AI content team optimizing for both traditional SEO and Generative Engine Optimization (GEO — citation by ChatGPT/Perplexity/Claude/Gemini/Google AI Overviews). Architecture is unchanged: Conductor → Department Heads → Specialists, Pattern B universal (deterministic Composio fetches → pure-LLM synth), post-approval deterministic dispatch. Persona roster (10 total): 5 Content + 2 Distribution + 3 Insight - Content: researcher / strategist / writer / geo-editor / formatter - Distribution: pipeline-reporter / slack-digest - Insight: feedback-tagger / theme-synthesizer / linear-filer Key pieces: - lib/personas/researcher/fetch.ts — Pattern B over Reddit/X/Firecrawl/Perplexity - lib/dispatch/providers.ts — ChannelVariant providers for 7 publish targets (GitHub PR, WordPress, Ghost, Notion, Reddit, LinkedIn, X). 4 confirmed Composio slugs, WordPress/Ghost TBD, X is BYO dev account. - lib/ui/components/approval-card.tsx — BlogDraft preview with channels checkbox group (founder picks destinations at approval time, dispatcher fans out one Formatter call per ticked target). - lib/shared/auth-configs.ts — Reddit/Twitter/WordPress entries flagged TBD pending Foundation registration. Replaces: - 6 GTM sales personas (researcher/qualifier/strategist/writer/scheduler/ brief-writer) — researcher/strategist/writer kept with content-domain re-prompts, qualifier/scheduler/brief-writer dropped. - CS + RevOps departments — replaced with Distribution. - OutreachDraft/CustomDeal/ActivationNudge/CRMUpdate artifact types — replaced with TopicResearchBrief/ContentOutline/BlogDraft/ChannelVariant/ PublishedArtifact. Verified: - pnpm typecheck + pnpm build green - Mock-mode dashboard renders DAG + BlogDraft approval gate with channels picker, no runtime errors - Live LLM harness: 7/10 personas pass on real Anthropic/Claude OAuth. Researcher produces "AI Cold Email Is Dead: Why Founders Who Publish Are Winning" as the recommended angle. Out of scope (next iteration): - BlogDraft → Formatter fanout dispatch handler in api/approvals/[id] - WordPress/Ghost Composio slug verification via _probe-mcp-tools - Strategist prompt budget (occasional 120s timeouts) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merges origin/main into the content pivot branch. The pivot's persona
roster (10 content personas, 3 departments) wins over main's parallel
additive blogs roster (which kept GTM + 6 new content personas alongside).
Resolved in favor of THIS branch:
- PersonaId / Department / ApprovalArtifactType / FanoutSource unions
- 10-persona registry, scopes, prompts (no GTM personas, no Sunny's 6)
- 3-department persona-meta + DAG layout
- Channels-checkbox UX on the BlogDraft approval card
- Connection-meta POPULAR list (publishing-first ordering)
Absorbed from main (additive, complementary):
- CompanyContext system (drizzle/migrations/0003_company_context.sql,
lib/state/company-context.ts, lib/state/gtm-metrics.ts,
lib/orchestrator/context-synth.ts, app/api/context/{route,refresh,
mock/context}/route.ts, lib/ui/components/{company-context-card,
edit-company-context-dialog}.tsx, lib/ui/hooks/use-company-context.ts)
- ICPProfile / GtmObjective / GtmMetric / CompanyContext types
- CompanyContextSchema + CompanyContextInputSchema
- companyContext Drizzle table
- README/INSTALL updates from main
CompanyContext threading into content personas is the next iteration —
the persona prompts already accept an optional `companyProfile` field
(see lib/state/workflows.ts:421-470). Slice-map for the 5 content personas:
- researcher: companyName, competitors, sourceUrl
- strategist: oneLiner, icp, positioning, valueProps, competitors, voiceTone
- writer: oneLiner, productDescription, valueProps, voiceTone (+ samples)
- geo-editor: oneLiner, valueProps, productDescription
- formatter: companyName, oneLiner, sourceUrl
Verified:
- pnpm typecheck green
- pnpm build green
- All conflict markers resolved
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PITCH.md is now the north-star file. Everything else (slide deck, demo
script, marketing copy, README) derives from it. If a teammate is unsure
where to point a sentence, this is north.
Locked positioning:
Hero: "Docs are for parsers. Blogs are for people. We turn one into
the other — for devtools founders whose engineering velocity
outruns their marketing."
Tagline: "One commit. Every channel. All your buyers."
ICP: Series A devtools companies whose docs ship daily and blogs
ship quarterly. Resend, Linear, Vercel, Stripe-shape.
Theme: "One for All" — literally restates one approval / all
destinations / one source / all audiences / one prompt /
the whole team executing.
Why this niche won the debate (research-driven):
- Anthropic-judged hackathons reward narrow + theatrical demos
(lawyer beat 500 devs at Cerebral Valley with permit-processing app)
- Profound's $1B Series C owns GEO observability narrative — we don't
fight there. We're generation, not measurement.
- Jasper/Copy.ai/Writer own broad "AI content" — we don't fight there.
- White space: Notra/PersonaBox/Docsie are adjacent; nobody has
multi-persona + multi-channel + founder-in-loop.
- Mintlify ($45M Series B, $500M val), Stainless ($25M Series A)
publicly naming docs-as-marketing-surface — category being established.
- Devtools = highest-WTP B2B niche; $4–12K/mo agency budgets prove buyer
pays.
Pivot history (locked here for future Claude sessions to find):
- 2026-05-08 — initial: AI GTM team (sales / CS / RevOps)
- 2026-05-09 — pivoted to AI content team (blog / GEO / multi-channel)
- 2026-05-10 — narrowed to devtools docs → multi-channel content. Locked.
CLAUDE.md updated to point Claude Code sessions at PITCH.md as the
first file to read.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the freeform-prompt textarea with a structured 3-input form
that locks the workflow to: company URL (voice + product context) +
docs URL (topic source) + destination (single output target). The
freeform path is preserved for backward compat (the persona harness
+ legacy callers still send a `prompt` string).
Why: the founder doesn't have to articulate the workflow — they hand
us the inputs and the team takes over. Voice samples are now
auto-extracted from the company's existing blog instead of pasted
manually at setup. Single destination = single approval gate (no
multi-channel fanout for v1).
Architecture
------------
Pre-fetch (Pattern B, deterministic TS):
fetchCompanyContextBundle(companyUrl)
Firecrawl scrape /, /blog (3 most recent posts), compute
VoiceFingerprint via 10 mechanical rules:
1. sentence length distribution (mean + stdev)
2. pronoun mode (we / i / neutral)
3. hook pattern (anomaly / contrarian / stat-led / announcement)
4. heading style (topical / question / named-concept)
5. code block frequency
6. opinion marker density
7. banned vocabulary scan (leverage / empower / unlock / etc.)
8. closing pattern (single-line-punch / wrapping-up / cta-only)
9. stat density
10. words-per-section ratio
fetchDocBundle(docsUrl)
Firecrawl scrape, return markdown.
Persona pipeline:
Researcher → Strategist → Writer → GEO-Editor → Formatter
Single-track. No fanout. Destination locked from form submit.
Approval flow:
One combined gate after Formatter (formatted-for-destination preview).
Persona prompt updates (research-driven from Composio / Inngest /
Linear / Stripe / Resend / Polar blog analysis):
- Word count target: 1,800–2,200 for blog-html (devtools mode);
250 for Reddit; 5–10 tweets for X
- 3 rhetorical moves locked: failure-mode-first / stat-anchored /
contrarian-or-anomaly opening
- Voice rules locked: pronoun mode never mixed; banned marketing
verbs (leverage/empower/unlock/seamless/robust); aggressive
sentence-length variation when source company does it
- Per-destination shape: H2 count, code block count, closing pattern
New files
---------
lib/personas/researcher/company-fetch.ts
lib/ui/components/run-input-form.tsx
Modified
--------
lib/shared/types.ts — Destination, RunWorkflowInput,
VoiceFingerprint, ALL_DESTINATIONS
lib/shared/schemas.ts — RunWorkflowRequestSchema accepts new
shape; backward-compat with `prompt`
lib/personas/registry.ts — researcher input accepts companyUrl /
docsUrl / destination; same for
strategist, writer, geo-editor
lib/personas/prompts/researcher.md — reasons over companyBundle +
docBundle, passes voice
fingerprint downstream
lib/personas/prompts/strategist.md — encodes word-count + section-
count targets per destination
+ 3 rhetorical moves
lib/personas/prompts/writer.md — voice rules locked from
fingerprint; per-destination
output shape
lib/personas/prompts/formatter.md — Reddit + X format rules
tightened from research
lib/state/workflows.ts — fetchResearcherBundleForInput dispatches
to new dual-bundle fetch when companyUrl
+ docsUrl present; injectItemContext
splices runInputs into every persona
input; parseRunInputsFromPrompt
recovers structured fields from the
synthesized prompt string
app/api/runs/route.ts — buildStructuredPrompt synthesizes a
Conductor-readable prompt from the form
payload; legacy email-extraction path
only fires when `prompt` is present
app/(dashboard)/page.tsx — swap PromptInput for RunInputForm
Verified: pnpm typecheck + pnpm build green; form renders with all 3
inputs + destination radio in mock mode.
Out of scope (roadmap):
- Multi-channel fanout from a single approval (architecture stays;
UI is single-destination)
- Visual theme extraction (colors, fonts) from company URL
- Outline-then-draft two-stage approval (collapsed to one)
- Founder-paste voice samples (replaced by auto-extraction)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…formatter
Three fixes from a real-LLM smoke that revealed the workflow could complete
"successfully" with no draft produced and no approval raised — the run
showed state="done" but the founder had nothing to review.
Fix 1: Fail loud on Firecrawl failures
When the researcher's Pattern B fetch can't reach the docs URL, throw
IntegrationFetchError with a founder-readable message ("Connect Firecrawl
on /connections, then retry"). Previously the synthesizer ran on an empty
docBundle, produced garbage, and the cascade swallowed the failure under
triggerRule: "all_done". Company-URL fetch failures stay soft (degraded
voice fingerprint, log warn, continue) since the doc content is the
load-bearing input.
Fix 2: Single-destination formatter materialization
The Conductor emits a generic `formatter` task with `fanoutOver: "channels"`
for multi-channel fanout. WorkContext returns empty channels[] for v1 so
expandPlan materialized ZERO formatter tasks, meaning the BlogDraft never
got formatted or surfaced for approval. Now: when the run has a structured
destination (3-input form path), expandPlan rewrites the formatter template
as a single non-fanout task with target = destinationToToolkit(destination)
(blog-html→github, reddit→reddit, x-thread→twitter). Multi-channel fanout
stays in the architecture for future use.
Fix 3: Approval-gate guard + IntegrationFetchError surfacing
After the workflow loop, if the plan included an approval-triggering
persona (geo-editor or formatter) but no approval row was created in
approval_requests, mark the run failed with a clear reason instead of
reporting "done" with nothing to review. Also catch IntegrationFetchError
at the top of runWorkflow and route it through markRunFailed without
re-throwing — the founder sees the actionable message in the run details.
Adds:
scripts/_inspect-runs.ts — debug helper to dump recent runs, approvals,
and event streams from the local DB for triage.
Verified:
pnpm typecheck + pnpm build green
Dashboard renders with no new errors
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Teammates pulling the branch no longer need to manually edit ~/.gmaestro/auth-configs.json — getAuthConfigId() falls back to this static map. Each person still needs to connect their own Firecrawl API key on the Composio side; this just tells the app which auth config to look up at runtime. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ration
The previous 120s ceiling was tuned for the GTM-era short outputs (cold
emails, scheduler payloads). The content pivot's writer + geo-editor +
formatter each generate or edit 1,800–2,200 word blog posts on Sonnet 4.6,
which routinely lands at 90–150s for the writer alone, plus 60–120s for
geo-editing and 30–90s for formatter.
Symptom: a real-LLM run on docs.composio.dev/toolkits/firecrawl finished
with state=failed and the new approval-gate-guard message ("Workflow
finished without producing a draft to approve"). Activity events showed
exact 120s gaps between writer→geo-editor→formatter persona_started
events — Promise.race kills each task right at the budget and the
all_done cascade lets pipeline-reporter / slack-digest keep going.
300s gives long-form personas room to land while still failing fast on
a genuinely hung model.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighter generation budget + faster demo. Three places updated: - strategist.md: word count table, estimatedWordCount example, section count math, blog-html override - writer.md: word count rule, blog-html shape header - run-input-form.tsx: destination card hint reddit (250w) and x-thread (5–10 tweets) unchanged — already short. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 4-turn cap was tuned for GTM-era personas (1 tool call + 1 synth turn). The content pivot's writer + geo-editor + formatter were hitting error_max_turns: writer especially, since drafting ~1,000 words sometimes requires the SDK to inject extra rounds for long completions or extended thinking — even without tools. Symptom from a real-LLM run on docs.composio.dev: writer: error_max_turns: Reached maximum number of turns (4) geo-editor + formatter: Unterminated string in JSON (truncated mid-output) 12 gives the SDK headroom while still failing fast on a looping model. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous writer prompt treated the docs as side context and the
outline as load-bearing. Real failure mode in production: writer
summarized the doc instead of translating it, output read as AI slop.
Rewrite leads with the killer framing — "translate, don't summarize"
— and reorders inputs by importance: docBundle.markdown FIRST (the
source material to internalize and re-explain), outline SECOND (the
skeleton), voiceFingerprint THIRD (the voice contract).
Added Translation rules (5 specific moves):
- Open with what changed, not what the doc IS
- Replace doc-style enumeration with narrative
- Show why the reader cares before the API
- Inline code only when load-bearing
- No hedge words ("might", "could", "may help")
Tightened failure handling:
- Doc bundle empty → refuse to draft, explicit [DOC FETCH FAILED]
placeholder pointing the founder to /connections (don't fabricate
a blog from nothing)
- Outline empty → fall back to doc-driven structure (still write)
- voiceFingerprint empty → default founder tone
Voice + rhetorical move sections kept (research-backed); just trimmed
and made more declarative.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Composio's docs (and most modern devtools docs — Mintlify, Docusaurus,
Vercel-hosted) are JS-rendered SPAs. Firecrawl's basic scrape was
returning the JS shell, not the rendered content — researcher saw
status: not_found and the diagnostic surface bailed the run.
Two args added to every FIRECRAWL_SCRAPE call:
- waitFor: 2500ms — give the page time to hydrate before extraction
- onlyMainContent: true — strip nav / footer / sidebar boilerplate
so the markdown is the actual article body
Per-fetch timeout bumped 8s/12s → 25s to accommodate the wait + scrape
+ network round-trip without flapping.
Symptom this resolves:
researcher: Couldn't fetch the docs URL via Firecrawl (status: not_found)
on https://docs.composio.dev/toolkits/firecrawl
Should now read the rendered Mintlify page cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two cuts to writer wall time: 1. Writer model dropped sonnet 4.6 → haiku 4.5. Sonnet was taking 5+ min on 1k-word drafts and blowing past the 300s timeout. Haiku is ~3× faster; quality on long-form is weaker but usable when the strategist delivers a concrete outline (which our prompts enforce). 2. The synthesized Conductor prompt in app/api/runs/route.ts still said "~2,000 words" even though strategist + writer + form UI were all updated to 1,000. Synced — the LLM now sees the consistent ~1,000 word target end-to-end. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…-editor/formatter
Switch back to default triggerRule:all_success so downstream personas
SKIP cleanly when an upstream fails, instead of running on empty input
for 5+ minutes producing [MISSING UPSTREAM DATA] placeholders.
Symptom: when researcher's Firecrawl fetch fails (Mintlify SPA, .md
URL not found, etc.), strategist correctly skips on all_success — but
writer/geo-editor/formatter were marked all_done and ran anyway. Each
spent its full 300s timeout budget trying to fabricate output from
nothing, then formatter died on truncated JSON. The user's read of
the dashboard: "writer running before researcher, strategist skipped
entirely" — symptom of researcher's silent failure (no persona_started
emitted from runtime when Pattern B throws) plus the all_done cascade
making downstream still pop on the activity feed.
The all_done cascade was a GTM-era hack ("ship at least a draft if
research failed"). Now redundant — IntegrationFetchError already loud-
fails Firecrawl issues at the dispatcher level, and the approval-gate
guard catches empty-output runs at the workflow level. No need to
silently cascade.
triggerRule:all_done now reserved for end-of-run reporter personas
(pipeline-reporter, slack-digest in distribution-mgr) which legitimately
should fire regardless of upstream content success.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Composio's docs (https://docs.composio.dev/toolkits/firecrawl) are Vercel-hosted Next.js client-side-rendered. The SSR'd HTML is just the JS shell; actual content renders post-hydration. 2.5s wasn't enough — Firecrawl returned shell HTML, length under our 100-char floor, status read as not_found. Per-fetch timeout bumped 25s → 40s to accommodate. 5s covers most slow-hydrating SPAs without making the dispatcher feel sluggish. If specific docs sites still return shell HTML, bump again or switch to a server-rendered URL for the demo. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ng script
Two issues blocking the docs-scrape persona:
1. Composio wraps Firecrawl's response, which itself wraps the result, so
markdown lives at r.data.data.markdown — the extractor only walked one
level and silently returned nothing on every successful scrape. Walk up to
3 levels in both lib/personas/researcher/fetch.ts and company-fetch.ts.
2. API_KEY toolkits like Firecrawl don't go through the OAuth Connect Link
flow, so connections created via Composio's dashboard playground end up
without a userId binding. tools.execute({ userId: "default" }) then
returns code 1810. Added scripts/connect-firecrawl.ts which uses
connectedAccounts.initiate() with config.val.generic_api_key to create a
properly-bound connection. Updated the IntegrationFetchError hint to
point at the script when status === "not_connected".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When raiseApproval inserts a pending approval, we now fire-and-forget a Slack DM via the existing sendApprovalDM() helper (which was wired but never called). Surfaces approval-needed events on Slack with a deep link to the dashboard's approval page — the founder can drive the workflow from either surface. Wiring: - New optional env var GMAESTRO_SLACK_CHANNEL controls the target channel/ user ID. Unset = no Slack notifications, dashboard remains the only approval surface. - sendApprovalDM was missing dangerouslySkipVersionCheck, so Composio was rejecting the SLACK_SEND_MESSAGE call with "Toolkit version not specified". Fixed. - New scripts/connect-slack-channel.ts helps the founder list their Slack channels, write the chosen target into ~/.gmaestro/.env, and send a test DM to verify the end-to-end path. Mock-mode polish: fetchResearcherBundleForInput now short-circuits when GMAESTRO_MOCK_PERSONAS=1, so a mock-mode demo run completes in ~3s instead of hanging on Firecrawl + Reddit + Perplexity calls that the mock persona would have ignored anyway. Diagnostic scripts (underscore-prefixed, not user-facing) added during the Firecrawl debugging session — kept for future Composio integration triage. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related fixes that came out of getting the Composio docs → blog pipeline working end-to-end:
What changed
Firecrawl
Slack approval DMs
Mock-mode polish
Diagnostics
Why
Before this PR, every content-pipeline run failed at the researcher step because Firecrawl returned empty markdown that downstream personas couldn't write from. The pipeline cascaded garbage and reported "workflow finished without producing a draft" 5+ minutes in. With these fixes, a real run produced a 2,140-char Firecrawl scrape of the Composio docs URL, and the dispatcher progressed through researcher → strategist → ContentOutline approval (verified live).
Test plan
Notes for reviewer
🤖 Generated with Claude Code