feat(datasource)!: invoke declared operations with host receipts

.claude/skills/refresh-generated-api/SKILL.md

@@ -21,11 +21,14 @@ git clone --filter=blob:none --no-checkout \
     https://github.com/codefly-dev/module-saas-starter.git starter
 cd starter
 git sparse-checkout set --no-cone 'module/services/accounts/proto'
-git checkout <ref>          # a commit SHA, so the result is reproducible
+git fetch origin tag <tag>
+git checkout --detach <tag> # use the published release tag when available
 ```
 
 Put it outside this repo's tree — a scratch directory, not a sibling path a
 `./...` pattern can pick up.
+Verify an annotated tag's target with `git rev-parse '<tag>^{commit}'`.
+For development before a release, use an exact PR-head commit instead.
 
 ## 2. Generate
 
@@ -41,7 +44,15 @@ which is what makes the descriptors carry this module's path. `out: gen` in the
 template is relative to `-o`, so `-o "$repo"` writes `$repo/gen`. Takes about a
 second.
 
-Two properties of that template matter:
+Three properties of that template matter:
+
+- **The inputs select the complete host-owned `saas/` schema tree.** The
+  vendored `codefly/` closure is import-only: its `go_package_prefix` resolves
+  to `github.com/codefly-dev/core/generated/go`. Require a compatible Core
+  version that provides those packages; validate the dependency floor against
+  SDK and consumer builds instead of copying the host's dependency generation.
+  Generating another copy of Core's descriptors here creates duplicate protobuf
+  registrations at initialization.
 
 - **`clean: true` wipes `gen/` first.** So a proto deleted upstream disappears
   here rather than lingering — which is how `AuditExportService` left in 0.1.0.
@@ -71,8 +82,10 @@ of the several green-making changes is the real fix.
 Three files move with the tree, and a tag that skips one claims a contract it
 does not carry:
 
-- **`SOURCE.txt`** — the ref you checked out, and the module version from
-  upstream's `module/module.package.codefly.yaml`.
+- **`SOURCE.txt`** — the release tag you checked out (or exact PR-head commit
+  before a release), and the module version from upstream's
+  `module/module.package.codefly.yaml`. CI resolves release pins through
+  `refs/tags/` so a same-named branch cannot select the source.
 - **`CHANGELOG.md`** — under the unreleased heading. `### Added` for new
   services, facades, and fields; `### Removed (breaking)` for anything the
   regeneration deleted, naming the symbols a consumer will fail to compile

.github/workflows/ci.yml

@@ -42,7 +42,7 @@ jobs:
   # gen/ must be exactly what buf.gen.yaml produces from the proto ref recorded
   # in SOURCE.txt. The recipe is the one in the refresh-generated-api skill:
   # this repo's buf.gen.yaml (plugin versions pinned there), run from
-  # module/services/accounts/proto of the source repo at the recorded commit.
+  # module/services/accounts/proto of the source repo at the recorded tag or commit.
   # clean: true in the template wipes gen/ first, so a stale, extra or edited
   # file all show up as a diff.
   gen-drift:
@@ -59,24 +59,28 @@ jobs:
         id: source
         run: |
           set -euo pipefail
-          line=$(grep -m1 -oE 'codefly-dev/module-saas-starter@[0-9a-f]{40}' sdk/SOURCE.txt || true)
+          line=$(grep -m1 -oE 'codefly-dev/module-saas-starter@(v[0-9]+\.[0-9]+\.[0-9]+|[0-9a-f]{40})$' sdk/SOURCE.txt || true)
           if [ -z "$line" ]; then
-            echo "::error file=SOURCE.txt::no 'codefly-dev/module-saas-starter@<40-hex sha>' ref found"
+            echo "::error file=SOURCE.txt::no 'codefly-dev/module-saas-starter@<release tag or 40-hex sha>' ref found"
             exit 1
           fi
           proto=$(grep -m1 -oE 'accounts public proto: [^)]+' sdk/SOURCE.txt | sed 's/^accounts public proto: //')
           if [ -z "$proto" ]; then
             echo "::error file=SOURCE.txt::no '(accounts public proto: <path>)' line found"
             exit 1
           fi
-          echo "sha=${line#*@}" >> "$GITHUB_OUTPUT"
+          ref=${line#*@}
+          if [[ "$ref" == v* ]]; then
+            ref="refs/tags/$ref"
+          fi
+          echo "ref=$ref" >> "$GITHUB_OUTPUT"
           echo "proto=$proto" >> "$GITHUB_OUTPUT"
           echo "source: codefly-dev/module-saas-starter@${line#*@} ($proto)"
       # The source repo is public, so the default token is enough.
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           repository: codefly-dev/module-saas-starter
-          ref: ${{ steps.source.outputs.sha }}
+          ref: ${{ steps.source.outputs.ref }}
           path: source
           sparse-checkout: ${{ steps.source.outputs.proto }}
           sparse-checkout-cone-mode: false
@@ -100,7 +104,7 @@ jobs:
         run: |
           set -euo pipefail
           if [ -z "$(git status --porcelain)" ]; then
-            echo "gen/ reproduces byte-for-byte from $(grep -m1 -oE 'module-saas-starter@[0-9a-f]{40}' SOURCE.txt)"
+            echo "gen/ reproduces byte-for-byte from $(grep -m1 -oE 'module-saas-starter@(v[0-9]+\.[0-9]+\.[0-9]+|[0-9a-f]{40})$' SOURCE.txt)"
             exit 0
           fi
           echo "::error::gen/ does not match a regeneration from the ref in SOURCE.txt. Never hand-edit gen/: regenerate it (refresh-generated-api skill), or correct SOURCE.txt if it records the wrong ref."

AGENTS.md

@@ -103,6 +103,8 @@ examples. Read it rather than re-deriving the API from the tree.
 - **Never hand-edit `gen/`, and never rewrite the module path with `sed`.** The
   file descriptors embed length-prefixed package strings; a text rewrite
   corrupts them and panics at `init()`. Regenerate (skill below).
+  Generate the host-owned `saas/` tree; Core's vendored Runnable imports resolve
+  to Core's Go dependency, never a second descriptor copy in this SDK.
 - **Every generated type reachable from a facade needs an alias in that
   package's `types.go`** — and a generated enum needs its constants
   re-exported too, because an alias carries the type and not the constants.

CHANGELOG.md

@@ -1,5 +1,114 @@
 # Changelog
 
+## Unreleased — callable sources
+
+- Reject unsupported HTTP response encodings before reading the body or
+  inferring an HTTP status: Declare/List return `DatasourceError` with no
+  receipt; Invoke/Lookup keep `OutcomeUnknown` and the original effect ID.
+  List returns `InputError` for an absent/unspecified host effect and
+  `DatasourceError` for an unknown future effect enum, with no partial list.
+- Fix cross-language parity at the error/receipt boundary: skip null and unknown
+  details without losing effect IDs; scope `SOURCE_OPERATION_OUTCOME_UNKNOWN`
+  to Invoke/Lookup; normalize UTF-8 charset, gzip case and BOM on Connect JSON
+  errors while retaining the gateway client and Connect decoder.
+- Add `DatasourceError{Receipt}` for invalid output on a valid COMMITTED receipt.
+  Receipt output comes from `receipt.output_json`; result output comes from the
+  top-level field. Both validate without discarding known commitment. Unknown
+  receipts ignore output; invalid receipt identity/status/time stays unknown.
+- Normalize absent listed schemas to `{}`; declaration still requires both
+  schemas before dispatch. Use the protobuf Timestamp JSON profile for receipt
+  times and rate-limit resets, rejecting year zero and excess fractional digits.
+- Add `datasource.Client.Invoke`, `Lookup`, `DeclareOperations` and
+  `ListOperations` on the existing gateway-bound Connect client, matching the
+  Python facade. Add operations, JSON output, receipts, UUIDv7 effect IDs and
+  per-call deadlines; the host alone owns replay and provider dispatch.
+- Add `ErrNotPermitted`, `ErrUnknownOperation`, `InputError`, `RateLimited`,
+  `ProviderRefused`, `OperationRefused`, `ErrEffectReused`, `ErrEffectNotFound`
+  and `ErrOutcomeUnknown`. The host's `SOURCE_OPERATION_OUTCOME_UNKNOWN`
+  reason retains the effect ID. `*OutcomeUnknown` wraps the original error for
+  `errors.Is`/`errors.As` and Connect diagnostics; callers must check
+  `ErrOutcomeUnknown` before applying transport retry rules. The SDK never retries.
+  Only `SOURCE_PROVIDER_REFUSED` maps to a provider refusal; its status comes
+  from the host's `provider_status` metadata (canonical HTTP 100–599, zero
+  when absent or invalid). `InputError.Pointer` preserves the host's first
+  `BadRequest.FieldViolation.field`. Host messages remain unchanged.
+- Add `OperationDeclarationRemoved` / `ErrOperationDeclarationRemoved` for
+  `SOURCE_OPERATION_DECLARATION_REMOVED`. It preserves the original effect ID
+  and `receipt_status` as `ReceiptStatus` and `Receipt.Status` (COMMITTED or
+  UNKNOWN; absent/unrecognized metadata becomes UNKNOWN). Recover with Lookup,
+  never with a fresh ID; the SDK never retries or invents receipt output.
+- Remove the old Lookup caveat: the host now recovers saved receipts after
+  declaration removal/replacement, under current source read authority.
+  Committed receipts survive; unresolved/expired stays unknown. Lookup NotFound
+  means no attempt marker exists.
+- Lookup misses return the original ID with `NOT_ATTEMPTED`; callers may
+  explicitly reinvoke with the same ID. Committed receipts include `Output`,
+  so recovery can retrieve output without another invoke. Receipt statuses are
+  `COMMITTED`, `UNKNOWN` and `NOT_ATTEMPTED`.
+- Every non-Connect HTTP failure after Invoke/Lookup dispatch, including 4xx,
+  429 and 500, returns `ErrOutcomeUnknown` with the original effect ID and
+  diagnostic cause. Decoded RPC errors retain their existing typed mappings.
+- Validate effect IDs and reject an expired deadline before dispatch with
+  `InputError`, no result and zero RPC calls. Explicit cancellation returns
+  `context.Canceled` with no result. Add operation descriptions and host-computed
+  digests to declaration/list round-tripping.
+- Document the call and retry contracts in `README.md` and package examples.
+- Invoke payloads use `input_json` / `output_json` strings containing exactly
+  one JSON object, preserving numeric precision. Declaration schemas retain
+  the host's `Struct` representation. Lookup sends only the effect ID; its
+  existing org/source arguments cannot override the host's derived authority.
+- Update `golang.org/x/tools` so the API boundary gate can read Go 1.27 export
+  data; the gate and its fixtures are unchanged.
+- Regenerate the full host API from released tag `v0.0.94`, recorded in
+  `SOURCE.txt` with module version `0.1.0`. The datasource bindings are unchanged
+  from the reviewed PR-head contract. This refresh creates no SDK release.
+  CI and the regeneration procedure now accept release-tag provenance.
+- Include the host's installation narrowing, organisation installation listing
+  and team grant/revoke RPCs, plus platform-admin module envelope reads/writes,
+  in the regenerated API. Module capability comments describe the host's
+  structured tenant-membership and idempotency refusals.
+- Include the runtime-facing `PruneSourceOperationReceipts` and
+  `LookupPruneSourceOperationReceipts` RPCs in generated bindings only;
+  no receipt retention methods are exposed by the caller facade.
+- Re-export the OAuth2 grant type/constants reachable from source reads. The
+  generation template emits all host-owned `saas/` schemas and imports Core's
+  Runnable descriptors from Core v0.3.41, which supplies the required package
+  while preserving consumer compatibility; plugin versions are unchanged.
+
+### Removed / changed (breaking, inherited host contract)
+
+- Host `v0.0.94` removes `MintModuleRegistration` and `MintSolutionRegistration`
+  RPCs, their procedure constants, and `ModuleMintRegistrationRequest/Response`
+  and `SolutionMintRegistrationRequest/Response`. Work Context mints now prove
+  workload identity with projected ServiceAccount tokens; there is no replacement
+  registration-credential mint RPC.
+- Generated `ModuleMintWorkContextRequest`, `ModuleMintOperationContextRequest`
+  and `ModuleMintSourceOperationContextRequest` lose `Secret`/`GetSecret` and
+  gain `ServiceAccountToken`/`GetServiceAccountToken`. The first two requests
+  also gain `OrganizationId` for shared-principal installation selection.
+  The gateway route-rejection fixture uses the current request field. The
+  hand-written broker sends the projected token as its sole Bearer credential,
+  and includes the organization selector on the first two REST mint bodies.
+- `moduleauthority.CurrentInstallation.SolutionIdentifier` is replaced by
+  `TargetID` and `BindingID`, reflecting the host's immutable authority
+  identities. The SDK validates the target UUID and nonempty binding.
+- Generated `Installation.SolutionIdentifier`,
+  `InstallSolutionRequest.SolutionIdentifier` and
+  `ModuleCurrentInstallationResponse.SolutionIdentifier` are removed in favor
+  of target/binding identity fields. Their `GetSolutionIdentifier` methods
+  disappear too.
+- Generated `PutSolutionRegistration` / `DeleteSolutionRegistration` RPCs,
+  procedure constants and request types (including the Put request's oneof
+  wrappers), `SolutionFrontendRegistration` and `SolutionBackendRegistration`
+  are removed. The host reconciles delivered presence; there is no writer RPC
+  replacement. Read contracts and new entitlement/target contracts are retained.
+- Generated `SolutionFrontendBinding.LeaseExpiresAt`,
+  `SolutionBackendBinding.LeaseExpiresAt`, `SolutionRegistration.RuntimeBoundary`
+  and their getters, and the
+  `SolutionRegistrationStatus_SOLUTION_REGISTRATION_STATUS_EXPIRED` constant
+  are removed by the same delivered-presence contract. No generated output was
+  edited or restored to retain the older API.
+
 ## Unreleased — exclude broker diagnostics
 
 - **Fix at the SDK error boundary; breaking.** Broker errors exclude all
@@ -16,21 +125,32 @@
 
 ## Unreleased — mint once: the projected ServiceAccount token replaces the module secret
 
-- **Breaking.** `moduleauthority.Credentials` is `{Prefix, Token}`: `Token` is a
-  `ProjectedTokenSource` that reads the module pod's projected ServiceAccount
+- **Breaking.** `moduleauthority.Credentials` is `{Prefix, Token, OrganizationID}`.
+  `Token` is a `ProjectedTokenSource` that reads the module pod's projected ServiceAccount
   token (audience `accounts`). `Secret` is deleted with the host's
   `MODULE_IDENTITY_SECRETS`. Every broker mint sends the token as
   `Authorization: Bearer`, read from its source before that mint (the kubelet
   rotates it); `X-Codefly-Module-Secret` is no longer sent.
+- `OrganizationID` selects the installation for module Work Context and
+  operation-context mints. Shared modules must supply it; tenant-pinned modules
+  may omit it or name their own tenant. Use a separate client per organization
+  so its cached module Work Context stays scoped. Source-context mints send only
+  `prefix` and `source_id`; their delegation determines the organization.
+- Every token source is trimmed and checked before dispatch: empty or oversized
+  (more than 64 KiB) tokens return `ErrInvalidCredentials` with zero broker calls.
+  The host requires audience `accounts` and lifetime at most 600 seconds. No
+  token path is defaulted. Tests cover file rotation and exact REST bodies on
+  all three mints; the host fake rejects missing, duplicate and non-Bearer
+  Authorization, and any legacy secret header as a regression guard.
 - Added `ProjectedTokenFile`, the file-backed source. Its method set is
   `sdk-go/workcontext`'s `ProjectedTokenSource`, so that implementation is
   accepted once this module moves off the pre-cutover `sdk-go/workcontext` pin.
 - Added `ErrExecutionUnestablished` for the broker's `503`: the host could not
   establish what the pod is running, nothing was decided, and a retry may
   succeed. `401` still maps to `ErrInvalidCredentials`.
-- `gen/` is unchanged: the broker client speaks the broker's REST contract, not
-  the generated request types, so the regeneration waits for the host change to
-  land and is its own refresh.
+- The broker client speaks the REST contract verified against
+  `gateway_modules.go` at released host tag `v0.0.94`; the generated request
+  types are consumed through the tagged API refresh above.
 
 ## Unreleased — module audit declarations and gateway authority
 
@@ -123,7 +243,7 @@ version and to signal the breaking removal below.
   - the three mints go to the gateway's module broker as JSON
     (`POST /modules/_work-context`, `/modules/_operation-context`,
     `/modules/_source-operation-context`) with `X-Codefly-Internal-Token` and
-    `X-Codefly-Module-Secret`;
+    `Authorization: Bearer <projected ServiceAccount token>`;
   - `ExchangeOperation` calls accounts' `authority` gRPC endpoint with the
     module Work Context in `x-codefly-work-context` and the internal token in
     `x-codefly-internal-token`, and still refreshes a rejected module Work
@@ -133,8 +253,8 @@ version and to signal the breaking removal below.
   token, gateway, gateway base URL or authority address, and on a plaintext
   seam to anything but `localhost` or a loopback IP unless the consumer sets
   `AllowInsecureHTTP` from its own mesh-protection assertion; and with
-  `ErrInvalidCredentials` on an empty prefix or secret. The broker client never
-  follows a redirect. The `connect.ClientOption` variadic is gone.
+  `ErrInvalidCredentials` on an empty prefix or missing token source. The broker
+  client never follows a redirect. The `connect.ClientOption` variadic is gone.
 
   Refusals change shape with the transport: a broker answer other than 200 is
   a `*BrokerError` (path, status, reason), wrapped with `ErrInvalidCredentials`