datasources: a provider is resolved from a composed service, not a Go import — object storage first #1268

What we are trying to do

Adding a datasource should not mean editing, compiling and releasing accounts.

pkg/business/datasources.go imports every provider package directly:

"accounts/pkg/datasource/apisource"
"accounts/pkg/datasource/connector"
"accounts/pkg/datasource/crawler"
"accounts/pkg/datasource/github"
"accounts/pkg/datasource/objectstore"

and declares the providers as four constants — github, api, crawler, upload. The connector model itself is already generic: pkg/datasource/connector/ states an eight-clause envelope, Interface is a closed set of five (files, pages, records, messages, events) with the comment "The set is closed: a new interface is a contract change. The set of providers is open", Descriptor carries config fields, credential modes, a readers model, a budget and Conformant with a mandatory Gap, and connectortest/ holds a connector to all of it.

So the model's open set is closed by its host's build. That is the whole of what this issue fixes.

Scope — one PR

Populate the connector registry from composed service endpoints rather than Go imports, and register object storage that way as the first provider.

• The registry resolves a provider from a declared service dependency, not from a compiled-in package. A composition without that service has no such provider in its catalog, and a source of it refuses by name — the behaviour feat(accounts): object-storage sources read through the composed object-storage service; the host's store client is deleted #1235 already establishes for object storage.
• objectstore registers through its composed service. feat(accounts): object-storage sources read through the composed object-storage service; the host's store client is deleted #1235 makes it a client of the composed object-storage service and deletes the host's own client.go, so the seam is half-built there and the proof costs one provider, not five.
• github, api and crawler keep working exactly as they do, still compiled in. Converting them is not in this PR.
• The import block above is deleted for objectstore only; the four provider constants stay until a provider contract is published.
• Credentials stay per call and are never persisted by a provider service (envelope clause 3, decisions/callable-sources.md). The per-credential budget stays host-side: it meters across every source and replica (datasource_credential_budgets), so it is shared state with a tenancy.
• The conformance suite runs against the provider as registered, not against a package.

Size: ~400 hand-written lines. Past ~500, stop and ask rather than finishing.

Ordering

Gated on #1235, which must merge first: it is the proof that a provider can be a composed service, and the content of the three PRs closed as contained in it (#1213, #1216, #1214's successor) lives only on its branch today.

What follows this issue, each its own record when it is reachable, not now:

1. publish a provider contract so a provider can be written in any language — this is where DatasourceItemReaders and DatasourceProvenance stop being accountsv1 types;
2. github moves out as a service — the honest test, 4 RPCs but ~1,700 lines of Go behind an app-setup handshake, a push receiver, an OAuth account link and a git client;
3. only then is "extract a datasource core, keep saas as a thin host" answerable with evidence rather than preference.

Why this shape

The surface is the largest in accounts: datasource.proto answers 34 RPCs, against 26 for authorization.proto, 25 for platform_admin.proto and 13 for organizations.proto. Classified at 74e35f0b:

A host-independent core would be the first 16, under half the surface. Six are not datasource work at all.

Two templates for the rest already run in this platform, so neither is invented here:

• a provider as a composed service — obin-ai/module-document-store composes xlsx, docx, pdf, libreoffice, parse and object-storage as six separately published agents, each pinned by release in service.codefly.yaml and dialed through declared dependencies with no address written down;
• one core, thin hosts — codefly-dev/authority's libraries/authority-core/go plus libraries/conformance/go, which feat(accounts): the host interfaces are decided by the shared authority core #1257 made accounts a second host of, re-implementing no rule, with the shared suite a required CI context.

Docs, in the same PR

• module/services/accounts/AGENTS.md — the datasource section, on how a provider is resolved.
• pkg/datasource/connector/CONFORMANCE.md — registration through a service rather than a package.
• module/services/README.md and DEPLOYMENT_TOPOLOGY.md if the service graph changes.

Reuse, and what is deliberately not reused

Reused: the connector envelope, Descriptor, the registry's validation, connectortest/, connector.Budgeted, and #1235's composed-service client. Not reused: nothing new is written that one of these already does — if this PR grows a second mechanism for provider resolution, it is wrong.

Fix or hack

A fix at the owner: the host stops owning the set of providers. No stopgap is expected; label one if it appears.

Context

• Investigation: obin-ai/handbook design/datasource-module.md (branch buckets, not yet merged) — the measurements above, both templates with their commits, the four host-coupled concerns, and the open question of whether the capability stays saas-only.
• Owner's framing (2026-10-27): "saas knows way too many datasources, so we should create a new module: datasource … would be great if this module is generic and different implementations can add custom behavior." The owner chose to prove the seam in saas first and leave the new-repo question open until after step 1 above.
• Related, not superseded: Object-storage datasource: the contract says a client offers and flags providers by the catalog, no client reads it, and the connector that exists cannot take a new source (no cursor, no push, no admission) #1211 (the bucket provider), A connected source rotates its credential in place: same source, same collection, verified by a read before it replaces the old — today a rotated key is a different source and a full re-ingest #1225 (credential rotation in place), A receiver for bucket notifications, so a change in a customer bucket the service does not front enqueues a change set of exactly the delivered keys instead of waiting for the daily reconcile #1226 (a bucket notification receiver), Callable sources: connecting a source into a new collection grants the connector datasource:invoke at its boundary #1102 (grants at connect).

🤖 Generated with Claude Code