Feature/security architecture#27
Open
jackm43 wants to merge 103 commits into
Open
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a check workflow that runs npm run check and npm test on every push and pull request. Disable remote bindings in the vitest pool so the suite runs hermetically without Cloudflare credentials; the tests already mock the AI binding, and the remote proxy session was the only thing requiring auth. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pin actions/checkout and actions/setup-node to full commit SHAs so a compromised tag cannot change what CI runs. Add Dependabot weekly update checks for npm and github-actions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
POST /gateway/start and GET /gateway/health previously authenticated with the raw Discord bot token, making the highest-privilege Discord credential double as an API password. Authenticate with a dedicated GATEWAY_CONTROL_TOKEN secret instead, failing closed with 401 when the secret is unset. Operators must run wrangler secret put GATEWAY_CONTROL_TOKEN and add the matching 1Password field used by deploy.sh. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/bicture buffered model-returned image URLs with no size cap and no timeout. Cap downloads at 25 MB via a content-length precheck and a post-download byteLength check, mirroring ragjam's audio handling, and abort the fetch after 30 seconds via AbortSignal.timeout. Add the same timeout to ragjam's audio download, which had the cap but no timeout. Violations throw so the existing error handling replies with the failure message. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every AI ingress (/ask, /bicture, /ragjam, gateway mentions, and tracked-thread replies) now runs a shared pre-flight guard in src/limits.ts before any model call or enqueue. It enforces an hourly request limit (AI_RATE_LIMIT_PER_HOUR, default 20) via the new rag_ai_requests table and a trailing-24h spend budget (AI_DAILY_BUDGET_USD, default 1.00) summed from rag_ai_spend_events. Spend events still pending cost reconciliation count as zero, and the guard deliberately fails open on D1 errors: availability over enforcement for a guild toy bot. The /rag command family stays unlimited. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Delete unused withTimeout and runChatModel from ai.ts, and unused CONFIG_DEFAULTS/ConfigKey/isConfigKey from config.ts. Drop the ignored _env parameter from loadConfig and update call sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Share one interaction-based getInvokerDisplayName in rag-utils.ts across ask/bicture/ragjam, move the duplicated errorDetails into logger.ts, and export MAX_DISCORD_MESSAGE_LENGTH (1900) from types.ts. Ragjam's full 2000-char cap is renamed DISCORD_MESSAGE_HARD_LIMIT to keep the two limits distinct. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the triplicated ctx.waitUntil + editOriginalInteractionResponse + catch-and-apologise pattern in rag/ask/bicture with a single handleDeferredInteraction helper in commands/deferred.ts. The helper guards missing interaction credentials (rag keeps its synchronous fallback), supports file attachments, and logs with per-command context. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add runTrackedChatCompletion / runTrackedWebSearchCompletion in tracked-ai.ts that create the spend source id, attach gateway metadata, and record the spend event around each completion. Rewire the three near-identical blocks in processAiQueueMessage, plus runAskCommand and generateThreadTitle. Also inline the buildConversation wrapper, which duplicated buildNormalThreadConversation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a nullOnError option to discordJsonRequest and rebuild fetchChannelMessages, fetchMessage, and fetchUsername on it instead of hand-rolled fetch + ok-check + json parsing. postChannelMessage keeps returning the raw Response since callers inspect status. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the 10-branch if-chain in handleInteractionRequest with a commandHandlers record keyed by command name, normalizing handler signatures with thin arrow wrappers. discord.js is only imported by scripts/register-commands.ts, never by the worker, so it belongs in devDependencies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
src/mention.ts mixed mention parsing, thread persistence, conversation building, and queue consumption. Split into four cohesive modules: - src/threads.ts: the rag_ai_threads D1 store - src/conversation.ts: conversation/context building and thread titles - src/consumer.ts: AI queue message processing - src/mention.ts: mention parsing and gateway entry only getMessageAuthorDisplayName lives in mention.ts (not conversation.ts) because handleGatewayMessageCreate needs it and conversation.ts imports stripMentionTokens from mention.ts; keeping it there avoids a cycle. Pure moves with mechanical import updates; no behavior change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test/index.test.ts (~3.2k lines) covered every area of the worker in one file. Split into per-area suites: - test/http-boundary.test.ts: routing, Discord signature verification, and gateway control auth (19 tests) - test/commands-ai.test.ts: /ask, /bicture, /ragjam (13 tests) - test/commands-rag.test.ts: the rag command family, /ragspend, and the spend worker (14 tests) - test/gateway-mention.test.ts: mention parsing, gateway message handling, the AI queue consumer, and the gateway durable object (21 tests) - test/limits.test.ts: the AI usage guard (11 tests) Shared setup (request signing, mock env builder, DB mock) moves to test/helpers.ts. Test bodies are unchanged; only imports were updated, and the unused DiscordGateway import was dropped. Still 78 tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every Cloudflare Queue message (AI jobs and spend jobs) now crosses the
wire as a Cap'n Proto EventEnvelope (src/contracts/envelope.capnp)
carrying version, type, id, occurredAt, source, actor, optional guildId,
and a union payload for thread_start/thread_reply/channel_reply/ragjam
and spend jobs. Serialisation uses capnp-es (pure-JS runtime, works in
workerd); the generated src/contracts/envelope.ts is committed so CI and
deploys never need the native capnp compiler. Regenerate with
`npm run contracts:build` (requires `capnp`, e.g. `brew install capnp`).
On top of the wire format, src/contracts/validate.ts enforces value
constraints Cap'n Proto cannot express: snowflake ids must match
/^\d{17,20}$/, free text (prompt/lyrics) is capped at 4000 chars,
usernames at 100. Validation runs at both encode (producer) and decode
(consumer) - zero trust between hops. isAiJob in validation.ts is
replaced by the stricter contracts decode path.
Decode failures return null; consumers log ai_job_invalid /
ai_spend_job_invalid and ack, mirroring the previous invalid-job path.
decode also pre-validates the capnp frame header before parsing because
capnp-es sizes its segment list from attacker-controlled bytes and can
be made to allocate unbounded memory otherwise.
Note: JSON messages already in flight during the deploy that ships this
will fail decode and be acked with a warn log - acceptable one-time
loss.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/ask no longer runs its whole AI pipeline inline in ctx.waitUntil. The interaction handler now only: checks usage limits, derives the thread title from the prompt (sanitizeThreadTitle fallback - no paid title model call), creates the thread, records it in rag_ai_threads, edits the deferred response with the thread link immediately, and enqueues an "ask" job through the Cap'n Proto event envelope (new ask union member in the schema; generated code regenerated). The queue consumer handles the ask job: builds the ask conversation, preserves the shouldUseAskWebSearch heuristic, runs the tracked completion, sanitizes, posts into the thread, and records the interaction like other kinds. If the model call fails, a failure notice is posted into the thread. thread_start jobs also use the prompt-derived title now. generateThreadTitle and its tracked-title spend path are deleted - one fewer model call per thread everywhere. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/rag previously ran a D1 batch (INSERT event + UPSERT totals) followed by a separate SELECT of rag_count; /undorag did the same after its DELETE + UPDATE batch. The upsert/update now carry RETURNING rag_count and the count is read from the batch results, saving one D1 round trip per command. Verified RETURNING works inside batch statements against workerd's D1 in the vitest pool; the shared DB mock's batch() now returns per-statement results. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
createBoundaryClient(policy) returns a fetch-shaped client that enforces
an egress policy before any network I/O: https-only, exact-host
allowlist (or wildcard for media), credential attachment so callers
never handle tokens, a default AbortSignal.timeout when the caller
passes no signal, and an optional response size cap enforced via
content-length precheck plus actual-byte verification while buffering.
Denials and failures are logged with a request context of
{identity, trustZone, method, host, outcome, status?} - never full URLs
with querystrings, and host only for egress-media.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
src/net/clients.ts builds per-identity boundary clients from Env, memoized per Env and constructed lazily so a missing token only fails the identity that needs it: - discord-rest (discord.com, Bot token) carries discordJsonRequest, postChannelMessage, and fetchBotRoleIds; botHeaders is gone. - discord-webhook (discord.com, no credential) carries editOriginalInteractionResponse, which now takes env for host policy and timeout despite needing no auth header. - ai-gateway (gateway.ai.cloudflare.com, cf-aig-authorization) carries both REST paths in src/ai.ts. - cloudflare-api (api.cloudflare.com, CLOUDFLARE_API_TOKEN) carries findGatewayLogCostMicros. - media-download (any https host, no credential, 25MB cap, 30s timeout) carries bicture's image download and ragjam's audio download; their hand-rolled caps and timeouts are deleted. Ragjam keeps its null-fallback on oversize via the typed response_too_large violation and bicture keeps its throw behavior. No module outside src/net/ reads DISCORD_BOT_TOKEN, CF_AIG_TOKEN, or CLOUDFLARE_API_TOKEN anymore (the DO's gateway IDENTIFY payload and Node-side scripts/ excepted), and no direct fetch remains outside src/net/. This is the enforcement point where a future per-worker split gives each worker only its own client and credential. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/bicture previously ran env.AI.run inline in the main worker's fetch path via ctx.waitUntil. It now defers and enqueues an encoded bicture job like /ragjam does, and the ai-jobs consumer generates the image, records the spend event, and edits the original interaction response with the attachment. Adds a BicturePayload kind to the Cap'n Proto envelope (regenerated with contracts:build) with the same encode/validate rules as ragjam. After this the main worker's fetch path does no AI work at all. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New ragbot-brain-worker (wrangler.brain.jsonc, src/brain-worker.ts) owns the ai-jobs queue consumer with the same batch/retry/DLQ settings, plus the D1, AI, and SPEND_JOBS producer bindings and the .md system-prompt rules it needs. The main worker loses its queue() handler, ai-jobs consumer, AI binding, and SPEND_JOBS producer (nothing in its fetch path records spend any more); it keeps the AI_JOBS producer. CF_AIG_TOKEN now belongs to the brain only; DISCORD_BOT_TOKEN is temporarily still needed by the brain because it posts to Discord until a responder worker takes over egress (documented in README). dev:all and deploy scripts include the brain config, and the README documents per-worker secrets and the one-time queue bootstrap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New ragbot-responder-worker (src/responder-worker.ts) is the single
choke point for posting as the bot, with a deliberate hybrid transport:
- Text-only replies (channel/thread posts, text interaction edits) go
through a new durable discord-outbox queue as encoded/validated
envelopes (reply.channel_message, reply.interaction_edit). Queue
messages cap at 128 KiB, which is why media does not go here.
- Media-bearing interaction edits (bicture image, ragjam audio) go over
a service-binding RPC (Responder WorkerEntrypoint) taking the encoded
envelope plus the attachment bytes: direct worker-to-worker, no
network exposure, and a retry would regenerate the media anyway.
The responder is the only worker whose Discord credential can write and
the only place final output policy runs: brains ship raw model text and
the responder applies sanitizeAiText, the MAX_DISCORD_MESSAGE_LENGTH
cap, and allowed_mentions {parse: []} on egress.
rag_ai_interactions.response_text still records the sanitized text (the
brain computes the same pure policy function for the record); status
'ok' now means handed to the outbox.
The brain keeps DISCORD_BOT_TOKEN for read-only REST (thread history,
replied-to context, bot roles) and thread creation — documented honestly
in the README trust-zone table. The main worker's deferred /rag and
/ask interaction edits keep the scoped, short-lived webhook path; this
boundary is documented too.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
handleGatewayMessageCreate (running inside the DiscordGateway DO) used to do a D1 thread lookup, a Discord REST role fetch, the usage-limits check, and denial posting. All of that moves to the brain: - DO path: any non-bot MESSAGE_CREATE that passes isDiscordMessage is encoded as a new message.received contracts kind (ids, length-capped content, author, mentions, mention_roles, reply metadata, guild id) and enqueued to ai-jobs. The only pre-filter is pure and local: both reply paths require a non-empty prompt after mention stripping, so those are skipped. Thread tracking lives in D1, which the DO deliberately cannot see, so everything else is enqueued and the brain filters. This trades queue volume for isolation — acceptable for a single small guild. - Brain: on message.received it does the thread lookup, bot-role fetch, mention resolution, and limits check, then processes the resolved thread_reply/channel_reply in-process (no re-enqueue). Denial notices go out via the discord-outbox. - The DO keeps only WebSocket lifecycle + IDENTIFY, payload validation, and encode+enqueue; it no longer uses D1 or Discord REST. It stays hosted in the main worker: moving a DO class between scripts needs a risky transfer migration, documented in the README as a possible future step (deliberate deviation from RECOMMENDATIONS.md section 1). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each worker now lives in its own folder as a deployable identity,
split by trust layer: workers/public/gateway (Discord ingress +
DiscordGateway DO) and workers/services/{brain,responder,spend}.
Worker entrypoints are thin — they only wire fetch/queue/RPC handlers
to package functions.
Business logic moves under packages/:
- contracts: queue envelopes, schema, validation, shared types
- net: boundary client + client factories
- discord: Discord REST module
- ai: model calls, tracked AI, prompts/config, ask mode, spend
- domain: commands, conversation, consumer, mention, threads,
limits, outbox, responder delivery, http verification
- logger: structured logging (leaf, used by net and everything above)
types/validation live in contracts and spend lives in ai so the
package graph stays acyclic (contracts <- net <- discord/ai <- domain
<- workers).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mirror the workers/packages layout under test/: worker-boundary and command-flow tests live in test/workers/, package-level tests in test/packages/ (contracts, net/boundary-client, domain/limits). Pure moves plus import-path fixes; still 114 tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace per-handler boilerplate in the gateway's interaction handling with declarative command specs in packages/domain/commands: - context.ts: builds a CommandContext (invoker, displayName, channel, guild, interaction credentials, options) once at the boundary. - registry.ts: CommandSpec type plus a shared pre-flight chain that runs option validation -> interaction credentials -> admin check -> usage limits, then dispatches by spec kind (enqueue / inline / deferred-inline). - specs.ts: the whole command surface as data; /bicture and /ragjam are pure buildJob enqueue specs, while /ask (REST thread creation before enqueue) and /rag (ban logic + deferred DB flow) keep run escape hatches on the shared context. - router.ts: looks up the spec and hands off to the registry. On the consumer side the if/else job-kind chain becomes a Record<kind, processor> map with one processor per AiJob kind. Behavior is unchanged; adds registry-level tests for unknown command, missing required option, admin denial, and missing interaction credentials (114 -> 118 tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The per-user hourly cap and per-user daily budget throttled legitimate heavy use, which the owner does not care about on a single friends-only server. Replace them with abuse-shaped guards: a per-user burst limit (default 8 requests per trailing minute, AI_BURST_LIMIT_PER_MINUTE) that catches floods and scripted spam while staying generous for humans, and a global trailing-24h budget across all users (default $10, AI_GLOBAL_DAILY_BUDGET_USD) as the wallet backstop if any account is compromised. Burst counting reuses rag_ai_requests; the global budget sums rag_ai_spend_events with no user filter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ALLOWED_GUILD_IDS (comma-separated guild snowflakes) now gates every ingress via packages/domain/guilds.ts: interactions get a friendly "home server" refusal (PING exempt for endpoint verification), the gateway Durable Object drops MESSAGE_CREATE events from non-allowed guilds and DMs before enqueueing, and the brain repeats the check on message.received as defense in depth between queue hops. When set, the gate fails closed — non-snowflake entries are dropped so a misconfigured value denies rather than allows. When unset, the gate allows but logs a warning once per isolate, so existing deploys keep working until the var is configured; set it to enforce. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
rag_command_bans previously only gated /rag. Move the active-ban lookup into a shared packages/domain/bans.ts and wire it into the registry pre-flight chain for every AI command (/ask, /bicture, /ragjam) and into the brain's message.received path, so mentions and tracked-thread replies from banned users are ignored (no notice on the gateway path). Slash-command denials mirror /rag's message: "You cannot use AI commands until <t:...>." The AI-path lookup fails open on D1 errors, matching the usage guard's availability-first stance. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Schema changes now flow through migrations/ applied with `wrangler d1 migrations apply`: 0001_initial.sql captures the full current schema with IF NOT EXISTS statements throughout, so the first apply against the existing production database is a safe no-op. The d1 scripts and deploy.sh (via npm run d1:migrate:remote) switch from the old `d1 execute --file=schema.sql` flow, all three DB-owning wrangler configs point migrations_dir at the shared directory, and schema.sql stays as a reference-only mirror. Migration-strategy decision: no 0002 ALTER TABLE for the rag_ai_interactions token-usage columns. SQLite has no ADD COLUMN IF NOT EXISTS, the prod table may or may not already have the columns (unknowable from the repo), and a recreate-and-copy would lose token values if they exist — either variant could strand deploys or drop data. recordAiInteraction therefore keeps its dual-INSERT fallback, now documented, until prod's shape is verified by hand. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Introduce the connectors MachinePrincipal (application zone, single registered service operation connector.invoke, verify-only receiver with a committed keyring entry) and the connector.invoke event envelope carrying the uniform phantom-token operations (grant / fetch / token / introspect / 3LO begin+complete). Adds ConnectorInvokeJob wire type + total validation, capnp struct + regenerated module, the fail-closed ConnectorResult surface, the CONNECTORS + CONNECTOR_STORE bindings and GitHub App secrets on Env, and the egress-connector boundary trust zone. No behaviour yet — machinery only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The core of packages/connectors: a uniform grant -> handle -> use model in which a real provider credential NEVER leaves the broker. A caller (identity token verified + Cedar-authorized) calls grant to exchange its identity for an opaque, caller-bound handle; fetch/token/introspect present that handle, which is re-verified against the caller on every use, re-authorized by Cedar, and resolves the credential server-side. authorizedFetch injects it and calls the provider through a per-connector host-allowlisted boundary client; every use is audit-logged with the full actor chain. Includes the four strategies (api_key, oauth2_client_credentials, oauth2_authorization_code 3LO seam, github_app reference impl with RS256 App-JWT minting supporting PKCS#8 and PKCS#1 keys), the declarative connector registry, the DO-backed grant + encrypted 3LO token stores with an in-memory test double, the per-isolate access-token cache, Cedar connector.* policies + brain->broker service-hop bootstrap permits, and the caller-side connectorsClient helper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The credential broker worker: a thin Connectors WorkerEntrypoint whose invoke method registers the manifest and delegates to handleConnectorInvoke, plus the ConnectorStore Durable Object backing the grant/token store. No route, no queue, no D1 — reachable only over the CONNECTORS service binding, so no internet-facing worker owns provider credentials. wrangler.jsonc binds SERVICE_REGISTRY (from ragbot-registry-worker) and the in-worker ConnectorStore DO, bundles the .cedar policies, and documents the GITHUB_APP_ID / GITHUB_APP_PRIVATE_KEY / SERVICE_PUBLIC_KEYS / CONNECTORS_TOKEN_ENC_KEY secrets. Dry-run clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
github-jwt.test.ts proves the App JWT is a real RS256 signature verifiable by the matching public key, with correct short-lived claims, and that BOTH PKCS#1 (wrapping path) and PKCS#8 (native) private keys import and sign. handler.test.ts proves the gate fails closed at every step: a token not addressed to the broker and a forged signature are rejected (401), a Cedar-unauthorized operation is refused (403) before any credential runs, an unknown connector is 404, and a handle presented by a service other than the one it was issued to is rejected (the phantom-token binding). A full grant -> authorizedFetch proves the credential is injected server-side, response headers are filtered (no Set-Cookie leak), the handle response carries no secret, and every use is audit-logged with the actor chain. +8 tests (216 total). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CONNECTORS.md documents the phantom-token model as the core pattern, the four kinds, how authn+authz apply to every operation, the grant/use API + audit entry shape, a step-by-step how-to-add-a-new-connector (api_key + oauth2 client-credentials examples) and how-to-add-a-caller, and the operator steps to make the GitHub App connector live (create App, generate key, install, set secrets, which worker binds the broker). Notes the Curity split-token variant and the Cloudflare Secrets Store upgrade path. STATUS.md gains a packages/connectors entry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Design-first (per decision): broker as the single inter-service token issuer via an exchangeServiceToken RPC, sender-side Cedar decision + token caching, receivers verify a cached broker JWKS, phased dual-verify migration from per-worker signing. No live changes until approved. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The seam through which the credential broker resolves every provider credential: a SecretsProvider interface (get/optional set), a secretsProvider(env, name) factory selecting a backend, and four backends — wrangler-env (default), cloudflare-secret-store, hashicorp-vault (KV v2 over a boundary client), and onepassword (1Password Connect, since the Node-only SDK cannot run on workerd). All fail closed: missing/unreachable -> null. Wires SECRETS_STORE / VAULT_* / OP_CONNECT_* structurally onto Env and adds egress-vault / egress-onepassword trust zones. Focused tests cover provider selection, wrangler-env fallback, and per-backend fail-closed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-worker Ed25519 signing is retained for service-to-service authentication (distributed blast radius, no broker on the hot path, already live). The broker is NOT made the central token issuer. The single-pattern goal is met at the authorization layer (Cedar authorizes every hop and every connector op). Adopted instead: built-in per-worker key rotation. Broker stays the authority for external provider credentials only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reorganise the credential broker so provider-specific code lives in ONE
cohesive file per provider (packages/connectors/providers/<name>.ts),
each declaring the kinds/flows it supports behind the strategy interface:
- github.ts — github_app (JWT crypto + installation exchange + inject)
- oauth2.ts — oauth2_client_credentials + oauth2_authorization_code
- api-key.ts — api_key
The generic broker infra (handler, registry, strategy table, grant store,
Cedar gate, audit) stays at the package top level and calls into providers;
providers never touch identity/Cedar.
Connectors now resolve their credential through the secrets-provider
module: a registry entry carries a {provider, ref} secret reference and
the strategy calls secretsProvider(env, ref.provider).get(ref.ref). The
github-app entry defaults to {provider:"wrangler-env", ref:"..."} so
behaviour is unchanged. A null resolution fails closed (the secret-
resolution gate); added a handler test proving grant denies (500) when the
secret is unprovisioned.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Update CONNECTORS.md for the new packages/connectors/providers/ layout
(one cohesive file per provider, generic infra apart at the top level),
the {provider, ref} secret-reference model and how a new connector is
added, and the four secrets backends. Records the 1Password spike result:
the official @1password/sdk cannot run on workerd (its sdk-core loads a
~10MB WASM synchronously from disk via fs.readFileSync at module load),
so the onepassword provider is implemented against 1Password Connect.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rospection
The connectors admin surface needs to know, per backend, whether a secret can
be written at runtime and whether the backend is configured at all — without
ever handling a secret value here.
- onepassword: implement `set` via the Connect item-patch API, reusing the
vault->item resolve walk. Replaces an existing field's value or appends a new
CONCEALED field; throws when the vault/item cannot be resolved (a runtime
write never conjures them). 1Password is now a runtime-writable backend.
- Add optional `configured()` to the SecretsProvider interface and implement it
for cloudflare-secret-store, hashicorp-vault, and onepassword (wrangler-env is
always configured — it is just `env`).
- Add `describeSecretsProviders(env)` -> [{name, writable, configured}], the
single source the admin surface reads. `writable` is presence of `set`
(cloudflare-secret-store stays read-only at runtime; wrangler-env is
deploy-time only), `configured` is the backend's env/binding presence.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds four Cedar-gated, audit-logged management ops to the credential broker,
carried on the existing connector.invoke envelope (no wire-schema change) and
running the SAME fail-closed pipeline as grant/fetch — authenticated identity
token, then a per-op connector.admin.* Cedar gate, then audit. No management op
ever returns a secret value.
- ConnectorOperation gains admin_list / admin_describe / admin_set_secret /
admin_providers; ConnectorResult gains the matching secret-free result bodies
(summaries, detail, provider capability, set-secret outcome). Validator
accepts them (broker-wide ops bear no locator; describe/set-secret bear a
connectorId).
- store.ts: a ConnectorConfigStore persists an admin-set {provider, ref}
override (a reference, never a value). The registry is immutable code, so the
handler overlays the override on EVERY credential-resolving path (grant/fetch/
token/authorize) — the change takes effect and survives.
- handler.ts: dev-proxy added as the admin caller (crypto allowlist); the admin
ops are gated by connector.admin.* against Connector::<id> (a "*" sentinel for
broker-wide list/providers). setConnectorSecret surfaces the per-backend
runtime write-capability HONESTLY: written (vault/1Password), provision_required
(cloudflare-secret-store, re-pointed but not written), rejected (wrangler-env
with a value — deploy-time only). Never fakes success; never logs the value.
- connectorsClient gains listConnectors/describeConnector/setConnectorSecret/
getSecretsProviders.
- Cedar: dev-proxy connector.admin.* permits (connectors.cedar) + the
dev-proxy -> connectors service hop (services.cedar), the broker's second caller.
Focused tests: the admin authz gate (dev-proxy allowed, brain denied) and the
full set-secret write-capability matrix. 230 tests pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dev-proxy becomes the connectors broker's admin caller, mirroring exactly
how it drives the gateway command path.
- Add the devProxyToConnectors service client (edge->application, same
DEV_PROXY_SIGNING_KEY) and the `connectors` manifest target; bind CONNECTORS
(-> ragbot-connectors-worker#Connectors) on the dev-proxy wrangler config.
- Factor the layered-auth gate (Better Auth session + Access-identity binding)
out of handleCommand into a shared authenticateSession, so the admin endpoints
authenticate IDENTICALLY to /api/command — one gate, no drift.
- /api/connectors (GET), /api/connectors/{id} (GET), /api/connectors/{id}/secret
(PUT, zod-validated), /api/secrets/providers (GET): each mints an on-behalf-of
token (sub = the acting Discord admin) and invokes the broker over the binding.
The secret value flows inward only and is never returned; set-secret maps the
broker outcome to HTTP (written/referenced 200, provision_required 202,
rejected 409).
- Reserve POST /api/connectors/{id}/grant, GET /api/connectors/{id}/installations,
GET|POST /api/connectors/{id}/callback as documented-but-501 so the contract is
stable ahead of the later grant/installations/callback work.
- Document it all in openapi.yaml and regenerate api-types; the PUT body is
compile-linked to the SetConnectorSecretRequest schema via `satisfies`.
npm run check clean, 230 tests pass, both configs dry-run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extends the single-page admin UI with a "Connectors" section shown after
sign-in, in the existing minimal vanilla-JS style (no external assets).
- Lists each connector with its kind/host, a configured/no-secret badge, the
resolving backend, and its flows (GET /api/connectors + /api/secrets/providers).
- Per connector: a form to choose a secrets backend (non-writable backends are
disabled for value entry and labelled "read-only at runtime" / "unconfigured"),
enter a reference/locator, and enter/rotate a secret value, PUT to
/api/connectors/{id}/secret with credentials:"include".
- Secret values are write-only: the field is cleared on submit and never
displayed; the result shows only the broker's outcome (written / referenced /
provision_required / rejected) and its detail message, and refreshes the badge.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- CONNECTORS.md: a new "admin surface" section — the connector.admin.* ops and
their Cedar actions, the config-store override that makes a set-secret survive,
the per-backend set-secret capability matrix (written / provision_required /
rejected), the /api/connectors/* + /api/secrets/providers HTTP API and its
reserved 501 paths, the callback (ragbot-dev.jsmunro.me/api/connectors/{id}/
callback) vs webhook (ragbot.jsmunro.me/webhooks/{id}) URL conventions, and the
operator bind/deploy steps. Updated the SecretsProvider snippet + backends
table: onepassword is now runtime-writable, set presence is the write signal.
- STATUS.md: refreshed the test count (230/26 files), and noted the broker admin
surface + the dev-proxy as its admin caller.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Webhooks now converge on a dedicated ingress worker at webhooks.jsmunro.me
with provider-prefixed routes /{provider}/{id} (e.g. /github/{id}), off the
Discord-interaction gateway. Updated the CONNECTORS.md URL convention.
- Add TODO.md capturing the webhook-ingress design (own subdomain, broker-side
connector.webhook.verify signature check, validate -> enqueue -> brain,
idempotency/replay) plus the remaining admin-surface work (reserved 501
endpoints, 3LO/Discord, bind CONNECTORS on the brain) and the separately
tracked tasks (brain rename, Task 12 remainder, dev-proxy hardening).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…transport Make the deployed refactor work end-to-end (/rag, /ask, @mentions): - Processor: InteractionSession Durable Object (workflows worker) runs deferred commands and edits the interaction response, where EGRESS + WORKFLOWS_SIGNING_KEY live; the gateway ingress verifies the Discord signature, returns the type-5 ack, and kicks the DO over a cross-script binding. - Trusted transport: capability-gated binding/queue hops carry claims-only (unsigned) tokens read without signature verification -- the binding graph is the authentication. mintClaims / decodeIdentityClaims + a "trusted" client and server mode, trusted-by-default for binding/queue; http still verifies. Removes the internal signing-key dependency across every service hop. - Gateway: only the current socket processes events and MESSAGE_CREATE is deduped by id, so a mention never double-replies. - deploy.ts: workflows before gateway, attest before registry (cross-script DO deps). Egress trust tests retargeted to the http path + broker caller allowlist. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the Ed25519 interaction-signature verification into @rag/ingress/discord as verifyDiscordSignature (returns the parsed body or null), so the platform webhook ingress can verify Discord without importing bot code — breaking the connectors -> bot dependency cycle ahead of folding Discord interactions into webhooks.jsmunro.me. The bot's verifyDiscordRequest now wraps it with its own interaction-shape validation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ssor (Phase 2)
Route Discord interaction callbacks through the neutral webhooks ingress
(webhooks.jsmunro.me/{clientId}/interactions): verify the Ed25519 signature by
client id, answer PING with type-1, else return the type-5 ack and kick the
InteractionSession DO — carrying no bot domain code. The DO's new run() owns the
FULL command dispatch, all-deferred: every outcome (inline reply, Cedar/limit/
option/guild/unknown rejection) becomes an edit of the deferred reply as
workflows. Enqueue commands run their job processor inline (workflows has no
AI_JOBS producer). Shared authorizeAndLimit keeps executeCommand (legacy gateway
path) and the new path on one authorization authority.
Wire DISCORD_INTERACTION_PUBLIC_KEYS (ragbot app id -> public key) + the
cross-script INTERACTION_SESSION binding on the webhooks worker. Gateway /discord
stays as fallback until the Discord portal URL is repointed.
check green (acyclic); 276 tests incl. dispatch conversions + ingress
PING/type-5/bad-sig/unknown-app.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e 3) Add packages/service-kit/identity/act-as-token.ts: mintActAs/verifyActAs plus buildActAsContext, a 60s envelope-bound EdDSA JWS proving one application may act as another. It names applications by open string ids (Discord/GitHub client ids, connector ids) rather than widening the closed MachinePrincipal union or its exhaustive policy maps — a parallel ActAsContext beside IdentityContext. Reuses token.ts's crypto/codec helpers (envelopeSha256, b64url*, EdDSA sign/verify), now exported, so there is one signing/parsing implementation. Tests: round-trip with envelope binding + 60s window; refused off its envelope, past expiry, at the wrong audience, with an unknown/pinned-mismatch issuer, and under a foreign key. check green (acyclic); 283 tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixes the mention double-reply at its root. The gateway held the mention idempotency guard as a 60s in-DO TTL, but the Discord websocket lives in a Durable Object and reconnects regularly (evictions, heartbeat misses, op 7/9); Discord's resume is at-least-once, so a MESSAGE_CREATE redelivered more than 60s after first receipt slipped past the window and replied twice. Move idempotency to the right layer: handleGatewayMessageCreate now kicks the InteractionSession DO keyed by idFromName(messageId) instead of enqueuing to AI_JOBS. The DO's claim() is a durable per-message guard (15-min TTL), so a redelivered mention addresses the same DO and is dropped however long after the first — regardless of reconnect churn. runMention resolves + replies via the shared processMessageReceivedJob. Mentions and slash-commands now share one processor DO; the gateway's 60s firstSeenMessage dedupe + sweep are removed. The ai-jobs message.received consumer is retained to drain in-flight jobs from the old path. check green; 283 tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Discord now delivers interaction callbacks to the webhooks worker
(webhooks.jsmunro.me/{clientId}/interactions), verified and tested end-to-end,
so the gateway's /discord route is dead traffic. Remove it from the route
bindings (application-bindings.ts, regenerated routes.ts/openapi.ts/openapi.yaml),
drop the discordSignature security scheme + discordInteractionGuard wiring, and
delete the opportunistic HTTP wake-up (interactions no longer hit the gateway;
the cron + the DiscordGateway DO watchdog alarm keep the socket up). The gateway
keeps the DiscordGateway DO, /gateway/start|stop|health, and the mention ws path.
Tests: the command dispatch, ban/limit denial, guild, and signature-boundary
behaviours that were asserted through the retired /discord entry are covered on
the all-deferred processor path (session-dispatch) and the neutral ingress
(webhooks-interactions); the obsolete gateway-entry tests are removed and the
key content assertions (undorag, ragspendboard, ban/limit messages) re-added on
the new path. check green; 248 tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add ApplicationAuthority, a Durable Object addressed by idFromName(appId) — one instance per registered application, that application's OIDC-style issuer. It owns the member services allowed to act as the application (a Cedar-gated set kept in the DO's own datastore, not enumerated in code) and its signing key, and mints short-lived, envelope-bound act-as tokens. It returns only tokens and public keys; the private key never leaves the DO. Self-hosted rather than a Cloudflare-managed OAuth flow because act-as is machine-to-machine (a service acting as an application, no human): every CF-managed OAuth path — Access-for-SaaS OIDC, workers-oauth-provider, service tokens — is built around a human authorization_code consent flow and offers no client_credentials grant. So the authority signs the assertion itself (identity/act-as-token.ts) and publishes its verifying key via jwks() so any verifier resolves it with off-the-shelf JWKS tooling. Kept the ApplicationRegistry directory singleton untouched (its list() enumerates every app) — a parallel authority DO sidesteps re-keying the directory to idFromName(appId) and the row backfill that would entail. This is purely additive: no existing trusted binding/queue hop gains any verification. - authority.ts: configure/addMember/removeMember/get/jwks + mint(intent, envelopeSha256), which Cedar-gates the member then signs the assertion. - services.cedar: service.act-as action, permitted when resource.members.contains(principal); members supplied as entities by the DO at mint time. No hard-coded pairs, mirroring invoke/exchange. - keyring.ts: actAsResolverFromEnv (string-issuer resolver over APPLICATION_PUBLIC_KEYS) + applicationPublicJwk publisher. - act-as-token.ts: buildActAsContext accepts a precomputed envelopeSha256, so the authority mints from the hash alone — the payload never reaches it. - registry worker: APPLICATION_AUTHORITY DO binding + migration v3; APPLICATION_SIGNING_KEYS / APPLICATION_PUBLIC_KEYS env types; DO exported through both barrels. check green (acyclic); identity suite green. No new tests this pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remove the manually-provisioned APPLICATION_SIGNING_KEYS secret path. The ApplicationAuthority DO now generates its own Ed25519 keypair on registration (configure/first mint) via crypto.subtle.generateKey and persists it to its durable storage; the private half never leaves the DO. Nothing is ever hand-created or committed. Verifiers resolve an issuer's key at runtime instead of from a keyring: actAsResolverFromAuthority fetches the issuer application's public JWKS from its authority DO over the binding, cached per isolate. The isolate is short-lived (it dies when the worker's job finishes), so the cache cannot outlive a rotation. APPLICATION_PUBLIC_KEYS remains only as an optional static override (tests, pinning); normally unset. - authority.ts: ensureSigningMaterial() generate-once-then-read from DO storage (single-threaded per instance, so race-free); jwks() serves the stored public key; mint() and configure() drive generation. - keyring.ts: actAsResolverFromEnv -> actAsResolverFromAuthority (runtime JWKS fetch over a structural APPLICATION_AUTHORITY binding double, env override kept). - contracts: drop APPLICATION_SIGNING_KEYS; APPLICATION_PUBLIC_KEYS now optional. check green (acyclic); identity suite green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…(Phase 3) Reshape the authority's mint gate from a manually-granted members set to an authorized-registration model grounded in attestation. A client becomes able to act as an application only through register(), which verifies the client's artifact carries a production attestation (a local match against the AttestationStore — no egress, per the pushed-and-stored attestation path) before recording it. So "who may act as this application" is exactly "what the repo, through CI attestation, says may" — not self-assertion. act-as and on-behalf-of are one primitive: an authorized mint request that differs only in the subject it carries. mint() gains an optional subject set as the token `sub` (the application itself when omitted); every mint still runs the Cedar service.act-as check, now driven by the registered (attested) clients. - authority.ts: register(request) attestation-gated via verifyArtifactAttestation; ClientRegistration record replaces the manual member; configure() no longer seeds members (rights come only through register); mint() carries subject. - removeMember kept for revocation; addMember removed. - contracts: APPLICATION_AUTHORITY binding stub gains register(), drops addMember. check green (acyclic); identity suite green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ase 3) Make act-as live on a real boundary: a sender can mint and attach an act-as token, and a receiver can verify it — both strictly opt-in, so no ordinary hop changes behaviour. Wire: ServiceMessage gains a backward-compatible actAsToken @2 :Text field (regenerated service.ts; old senders omit it, receivers that don't opt in ignore it). encode/decodeServiceMessage + wrapServiceMessage/parseServiceMessage thread it as optional. Send (client.ts): a new actAs option on createClient().to({ actAs: { appId, subject? } }) asks the APPLICATION_AUTHORITY DO (structural binding double, kept app-agnostic) to mint an envelope-bound token proving self may act as appId, and carries it beside the identity token. Fails the hop closed if the authority is unbound or refuses. subject sets on-behalf-of. Receive (server.ts): verifyActAs/requireActAs config verifies a carried token against the issuer application's JWKS (actAsResolverFromAuthority — runtime fetch), checking aud == self, expiry, and envelope binding, then attaches the verified ActAsContext to RequestContext.actAs; a present-but-invalid token denies. Absent by default and ignored unless opted in. check green (acyclic); 248 tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.