diff --git a/RELEASE_GATES.md b/RELEASE_GATES.md index 1c8abff68..4abbe76de 100644 --- a/RELEASE_GATES.md +++ b/RELEASE_GATES.md @@ -229,8 +229,10 @@ A successful image build and matching materials alone do not establish that a generated-file change belongs in the PR. Reproduce recipes using the CI-pinned CLI and the released agents selected by the service manifests. Every topology service must declare image coverage, even if its recipe is absent -from the checkout. Redis and Vault explicitly use the existing Codefly vendor -image audit; an unknown agent is an error. +from the checkout. Redis, Vault and object storage explicitly use the existing +Codefly vendor image audit — the object-storage agent runs its gateway and MinIO +images, both pinned by the agent, and builds nothing here; an unknown agent is +an error. Changes to the image contract force the full service graph through CI. Before each canonical build attempt, CI snapshots Buildx history. After the build, diff --git a/libraries/source-read-sdk/library.codefly.yaml b/libraries/source-read-sdk/library.codefly.yaml index efe323fc4..0171cc00f 100644 --- a/libraries/source-read-sdk/library.codefly.yaml +++ b/libraries/source-read-sdk/library.codefly.yaml @@ -12,7 +12,7 @@ sources: module: saas-starter service: accounts endpoint: authority - contract-digest: sha256:10a231ea430313364937808160138bca599ac012230ad70b7047eac27764836a + contract-digest: sha256:743b63bad9a546f105e994c754d845865e189465e6c72816e5785efe38ab7831 services: - ModuleCapabilitiesService generated-by: diff --git a/module/DATABASE_AUTHORITY.md b/module/DATABASE_AUTHORITY.md index 969a5a33c..5d6e35fe9 100644 --- a/module/DATABASE_AUTHORITY.md +++ b/module/DATABASE_AUTHORITY.md @@ -526,7 +526,7 @@ executable inventory and this table in the same change. | Scope | Relations | Required database boundary | |---|---|---| | `global` | `audit_event_types`, `authority_core_delegations`, `authority_core_grant_epochs`, `authority_core_installations`, `authority_core_module_revisions`, `authority_core_modules`, `authority_core_owner_aware`, `authority_core_person_epochs`, `authority_core_replay`, `authority_core_role_grants`, `authority_core_tombstones`, `bootstrap_state`, `data_retention_policies`, `datasource_credential_budgets`, `email_templates`, `feature_flags`, `hop_admission_keys`, `hop_admission_proofs`, `identity_providers`, `module_envelopes`, `plan_entitlements`, `plans`, `platform_admins`, `solution_delivery_documents`, `solution_generation_history`, `solution_host_bindings`, `solution_registrations`, `solution_targets` | No RLS; exact grants | -| `tenant` | `actor_chain_journal`, `actor_chain_revocations`, `api_keys`, `approval_decisions`, `approval_requests`, `audit_event_idempotency`, `audit_events`, `connector_credentials`, `dashboards`, `datasource_account_links`, `datasource_domains`, `datasource_group_bindings`, `codefly_effect_receipts`, `datasource_operation_attempts`, `datasource_oauth_consumed_states`, `datasource_source_operations`, `datasource_sources`, `delegation_grants`, `domain_events`, `entitlement_overrides`, `executable_artifact_approvals`, `github_app_installations`, `github_app_setups`, `installations`, `invitations`, `membership_integrity_findings`, `org_generic_settings`, `org_identity_providers`, `org_key_bindings`, `org_settings`, `organization_activations`, `organization_authorization_revisions`, `organization_members`, `organizations`, `principal_authorization_revisions`, `principals`, `record_shares`, `role_assignments`, `role_permissions`, `roles`, `scope_grants`, `scope_nodes`, `source_delegations`, `source_read_revisions`, `subscriptions`, `team_members`, `team_membership_quarantine`, `teams`, `usage_events`, `usage_totals`, `webhook_deliveries`, `webhook_subscriptions`, `work_context_replay` | Enabled and forced RLS with at least one policy | +| `tenant` | `actor_chain_journal`, `actor_chain_revocations`, `api_keys`, `approval_decisions`, `approval_requests`, `audit_event_idempotency`, `audit_events`, `connector_credentials`, `dashboards`, `datasource_account_links`, `datasource_domains`, `datasource_group_bindings`, `datasource_object_manifests`, `codefly_effect_receipts`, `datasource_operation_attempts`, `datasource_oauth_consumed_states`, `datasource_source_operations`, `datasource_sources`, `delegation_grants`, `domain_events`, `entitlement_overrides`, `executable_artifact_approvals`, `github_app_installations`, `github_app_setups`, `installations`, `invitations`, `membership_integrity_findings`, `org_generic_settings`, `org_identity_providers`, `org_key_bindings`, `org_settings`, `organization_activations`, `organization_authorization_revisions`, `organization_members`, `organizations`, `principal_authorization_revisions`, `principals`, `record_shares`, `role_assignments`, `role_permissions`, `roles`, `scope_grants`, `scope_nodes`, `source_delegations`, `source_read_revisions`, `subscriptions`, `team_members`, `team_membership_quarantine`, `teams`, `usage_events`, `usage_totals`, `webhook_deliveries`, `webhook_subscriptions`, `work_context_replay` | Enabled and forced RLS with at least one policy | | `user` | `client_authorization_codes`, `gdpr_requests`, `mfa_backup_codes`, `mfa_devices`, `mfa_login_transactions`, `notifications`, `onboarding_progress`, `resource_follows`, `sessions`, `user_consent_events`, `user_consent_preferences`, `user_identities`, `users`, `webauthn_ceremonies`, `webauthn_credentials` | Enabled and forced RLS with at least one policy | | `pre_auth` | `magic_links`, `waitlist_entries` | Enabled and forced RLS; fail-closed request policy, accessed only by the control-plane role | | `job` | `job_messages` | Enabled and forced RLS with at least one policy; no request relation grant — function-only scoped enqueue plus exact job-worker grants | diff --git a/module/DEPLOYMENT_TOPOLOGY.md b/module/DEPLOYMENT_TOPOLOGY.md index 8362f5bc5..f224e47a4 100644 --- a/module/DEPLOYMENT_TOPOLOGY.md +++ b/module/DEPLOYMENT_TOPOLOGY.md @@ -36,6 +36,7 @@ REST without a corresponding accounts endpoint, generation fails. | Caller | Target endpoints | Kubernetes ports | | --- | --- | --- | | `accounts` | `cache/read`, `cache/write` | TCP 6379 | +| `accounts` | `object-storage/grpc` | Codefly-assigned gRPC port | | `accounts` | `store/tcp` | TCP 5432 | | `accounts` | `vault/http` | TCP 8200 | | `auth-gateway` | `accounts/connect`, `accounts/rest`, `accounts/grpc` | TCP 8080, 9090 | diff --git a/module/contracts/api/accounts/authority/contract.binpb b/module/contracts/api/accounts/authority/contract.binpb index eaa01f6d1..8cfc99aa6 100644 Binary files a/module/contracts/api/accounts/authority/contract.binpb and b/module/contracts/api/accounts/authority/contract.binpb differ diff --git a/module/contracts/api/accounts/authority/proto/saas/accounts/v1/datasource.proto b/module/contracts/api/accounts/authority/proto/saas/accounts/v1/datasource.proto index 60efdaffa..c95696bd1 100644 --- a/module/contracts/api/accounts/authority/proto/saas/accounts/v1/datasource.proto +++ b/module/contracts/api/accounts/authority/proto/saas/accounts/v1/datasource.proto @@ -189,7 +189,48 @@ message UploadDatasourceConfig { // AWS-style access key id. The matching secret access key is the credential. string access_key_id = 5; // Upper bound on objects fetched per sync. Zero applies the connector default. + // Only objects that pass the filters below count against it. uint32 max_objects = 6; + // Which keys under the prefix are ingested. Each pattern is an RE2 regular + // expression matched against the object key with the prefix removed, + // unanchored and case-sensitive (anchor with ^ and $, fold case with (?i)). + // An empty include admits every key; an exclude match always wins. Applied + // when the prefix is listed, before anything is fetched, so an excluded + // object is never read and never counts against max_objects. An invalid + // expression is refused when the source is added or updated, never at sync: + // the call fails INVALID_ARGUMENT with a google.rpc.BadRequest violation, and + // a message that starts with the field's path, e.g. + // "upload.include_patterns[1]: …". + repeated string include_patterns = 7 [(buf.validate.field).repeated = { + max_items: 16 + items: { + string: { + min_len: 1 + max_len: 256 + } + } + }]; + repeated string exclude_patterns = 8 [(buf.validate.field).repeated = { + max_items: 16 + items: { + string: { + min_len: 1 + max_len: 256 + } + } + }]; + // Case-insensitive key suffixes, e.g. [".md", ".pdf"], intersected with the + // patterns. Empty admits every type. GitHub's file_extensions semantics. + repeated string file_extensions = 9 [(buf.validate.field).repeated = { + max_items: 32 + items: { + string: { + min_len: 2 + max_len: 33 + pattern: "^\\.[A-Za-z0-9][A-Za-z0-9._-]*$" + } + } + }]; } // Datasource is one connected external datasource. It is org-scoped and carries @@ -437,6 +478,27 @@ message UpdateSourceRequest { // sole reason that the source had reconciled since the form was opened. What // an editor needs guarded is what they were looking at. SourceFetchScope expected = 6 [(buf.validate.field).required = true]; + // An object-storage source's key patterns (see UploadDatasourceConfig), + // replaced whole like every field here. Its file_extensions is field 5; its + // branch and paths must be empty. Refused for a github source. + repeated string include_patterns = 7 [(buf.validate.field).repeated = { + max_items: 16 + items: { + string: { + min_len: 1 + max_len: 256 + } + } + }]; + repeated string exclude_patterns = 8 [(buf.validate.field).repeated = { + max_items: 16 + items: { + string: { + min_len: 1 + max_len: 256 + } + } + }]; } // SourceFetchScope is what a source fetches — the three values UpdateSource @@ -467,6 +529,17 @@ message SourceFetchScope { max_len: 33 } } + }]; repeated string include_patterns = 4 [(buf.validate.field).repeated = { + max_items: 16 + items: { + string: {max_len: 256} + } + }]; + repeated string exclude_patterns = 5 [(buf.validate.field).repeated = { + max_items: 16 + items: { + string: {max_len: 256} + } }]; } diff --git a/module/contracts/api/accounts/connect/contract.binpb b/module/contracts/api/accounts/connect/contract.binpb index eaa01f6d1..8cfc99aa6 100644 Binary files a/module/contracts/api/accounts/connect/contract.binpb and b/module/contracts/api/accounts/connect/contract.binpb differ diff --git a/module/contracts/api/accounts/connect/proto/saas/accounts/v1/datasource.proto b/module/contracts/api/accounts/connect/proto/saas/accounts/v1/datasource.proto index 60efdaffa..c95696bd1 100644 --- a/module/contracts/api/accounts/connect/proto/saas/accounts/v1/datasource.proto +++ b/module/contracts/api/accounts/connect/proto/saas/accounts/v1/datasource.proto @@ -189,7 +189,48 @@ message UploadDatasourceConfig { // AWS-style access key id. The matching secret access key is the credential. string access_key_id = 5; // Upper bound on objects fetched per sync. Zero applies the connector default. + // Only objects that pass the filters below count against it. uint32 max_objects = 6; + // Which keys under the prefix are ingested. Each pattern is an RE2 regular + // expression matched against the object key with the prefix removed, + // unanchored and case-sensitive (anchor with ^ and $, fold case with (?i)). + // An empty include admits every key; an exclude match always wins. Applied + // when the prefix is listed, before anything is fetched, so an excluded + // object is never read and never counts against max_objects. An invalid + // expression is refused when the source is added or updated, never at sync: + // the call fails INVALID_ARGUMENT with a google.rpc.BadRequest violation, and + // a message that starts with the field's path, e.g. + // "upload.include_patterns[1]: …". + repeated string include_patterns = 7 [(buf.validate.field).repeated = { + max_items: 16 + items: { + string: { + min_len: 1 + max_len: 256 + } + } + }]; + repeated string exclude_patterns = 8 [(buf.validate.field).repeated = { + max_items: 16 + items: { + string: { + min_len: 1 + max_len: 256 + } + } + }]; + // Case-insensitive key suffixes, e.g. [".md", ".pdf"], intersected with the + // patterns. Empty admits every type. GitHub's file_extensions semantics. + repeated string file_extensions = 9 [(buf.validate.field).repeated = { + max_items: 32 + items: { + string: { + min_len: 2 + max_len: 33 + pattern: "^\\.[A-Za-z0-9][A-Za-z0-9._-]*$" + } + } + }]; } // Datasource is one connected external datasource. It is org-scoped and carries @@ -437,6 +478,27 @@ message UpdateSourceRequest { // sole reason that the source had reconciled since the form was opened. What // an editor needs guarded is what they were looking at. SourceFetchScope expected = 6 [(buf.validate.field).required = true]; + // An object-storage source's key patterns (see UploadDatasourceConfig), + // replaced whole like every field here. Its file_extensions is field 5; its + // branch and paths must be empty. Refused for a github source. + repeated string include_patterns = 7 [(buf.validate.field).repeated = { + max_items: 16 + items: { + string: { + min_len: 1 + max_len: 256 + } + } + }]; + repeated string exclude_patterns = 8 [(buf.validate.field).repeated = { + max_items: 16 + items: { + string: { + min_len: 1 + max_len: 256 + } + } + }]; } // SourceFetchScope is what a source fetches — the three values UpdateSource @@ -467,6 +529,17 @@ message SourceFetchScope { max_len: 33 } } + }]; repeated string include_patterns = 4 [(buf.validate.field).repeated = { + max_items: 16 + items: { + string: {max_len: 256} + } + }]; + repeated string exclude_patterns = 5 [(buf.validate.field).repeated = { + max_items: 16 + items: { + string: {max_len: 256} + } }]; } diff --git a/module/contracts/api/catalog.codefly.json b/module/contracts/api/catalog.codefly.json index 2f9b78e41..192675738 100644 --- a/module/contracts/api/catalog.codefly.json +++ b/module/contracts/api/catalog.codefly.json @@ -10,7 +10,7 @@ "kind": "protobuf", "package": "saas.accounts.v1", "path": "contracts/api/accounts/authority/contract.binpb", - "digest": "sha256:10a231ea430313364937808160138bca599ac012230ad70b7047eac27764836a", + "digest": "sha256:743b63bad9a546f105e994c754d845865e189465e6c72816e5785efe38ab7831", "services": [ { "name": "ModuleCapabilitiesService", @@ -66,7 +66,7 @@ "kind": "protobuf", "package": "", "path": "contracts/api/accounts/connect/contract.binpb", - "digest": "sha256:10a231ea430313364937808160138bca599ac012230ad70b7047eac27764836a", + "digest": "sha256:743b63bad9a546f105e994c754d845865e189465e6c72816e5785efe38ab7831", "services": [ { "name": "ArtifactService", diff --git a/module/deployment/README.md b/module/deployment/README.md index 0481cdf03..93a20a1b1 100644 --- a/module/deployment/README.md +++ b/module/deployment/README.md @@ -385,6 +385,7 @@ services: - name: cache - name: frontend - name: marketing + - name: object-storage - name: store - name: telemetry # no `vault`: KEY_SERVICE_BACKEND=kms with diff --git a/module/deployment/generated/service-topology.json b/module/deployment/generated/service-topology.json index 7216daef9..d6ad174e0 100644 --- a/module/deployment/generated/service-topology.json +++ b/module/deployment/generated/service-topology.json @@ -38,6 +38,12 @@ "write" ] }, + { + "service": "object-storage", + "endpoints": [ + "grpc" + ] + }, { "service": "store", "endpoints": [ @@ -136,6 +142,17 @@ } ] }, + { + "name": "object-storage", + "endpoints": [ + { + "name": "grpc", + "api": "CODEFLY_API_GRPC", + "visibility": "ENDPOINT_VISIBILITY_PRIVATE", + "port": 9090 + } + ] + }, { "name": "store", "endpoints": [ @@ -214,6 +231,12 @@ "ports": [ 443 ] + }, + { + "service": "object-storage", + "ports": [ + 443 + ] } ] } diff --git a/module/module.codefly.yaml b/module/module.codefly.yaml index 41914927e..795ca57ca 100644 --- a/module/module.codefly.yaml +++ b/module/module.codefly.yaml @@ -49,5 +49,6 @@ services: - name: cache - name: frontend - name: marketing + - name: object-storage - name: store - name: vault diff --git a/module/module.package.codefly.yaml b/module/module.package.codefly.yaml index d538e4f7f..8f9a3cb0a 100644 --- a/module/module.package.codefly.yaml +++ b/module/module.package.codefly.yaml @@ -18,12 +18,12 @@ services: kind: protobuf package: saas.accounts.v1 path: contracts/api/accounts/authority/contract.binpb - digest: sha256:10a231ea430313364937808160138bca599ac012230ad70b7047eac27764836a + digest: sha256:743b63bad9a546f105e994c754d845865e189465e6c72816e5785efe38ab7831 - endpoint: connect kind: protobuf package: "" path: contracts/api/accounts/connect/contract.binpb - digest: sha256:10a231ea430313364937808160138bca599ac012230ad70b7047eac27764836a + digest: sha256:743b63bad9a546f105e994c754d845865e189465e6c72816e5785efe38ab7831 - name: auth-gateway endpoints: [grpc, rest] api-contracts: @@ -38,6 +38,8 @@ services: endpoints: [http] - name: marketing endpoints: [http] + - name: object-storage + endpoints: [grpc] - name: store endpoints: [tcp] - name: vault diff --git a/module/services/README.md b/module/services/README.md index ad77f5d40..d898179a5 100644 --- a/module/services/README.md +++ b/module/services/README.md @@ -9,6 +9,12 @@ wrong, and nothing enforces it. `marketing` and `frontend` are separate Next.js applications: marketing owns public apex/`www` content, while frontend remains the authenticated product behind `auth-gateway`. +`object-storage` is the composed object-storage gateway +(codefly-dev/service-object-storage). The host keeps no store client of its +own: accounts reads every object-storage datasource through it, under a +per-call binding of the source's bucket and credential, and the gateway owns +the store protocol and the public-destination guard — see +`accounts/AGENTS.md`. Run the complete local dependency graph from `module/`: diff --git a/module/services/accounts/AGENTS.md b/module/services/accounts/AGENTS.md index a75265b4d..b917fe49f 100644 --- a/module/services/accounts/AGENTS.md +++ b/module/services/accounts/AGENTS.md @@ -1173,6 +1173,43 @@ The transport underneath — h2c rather than server TLS, the resolved origin, an the key set a consumer verifies host-signed credentials against — is [../../INTERNAL_TRANSPORT.md](../../INTERNAL_TRANSPORT.md). +## An object-storage source is a prefix and a key filter + +An `upload` ("Object storage") source reads one bucket under one `prefix`, and +`UploadDatasourceConfig` narrows it further: `include_patterns` and +`exclude_patterns` are RE2 expressions over the object key with the prefix +removed (unanchored, case-sensitive; exclude wins; an empty include admits +everything), and `file_extensions` intersects them with GitHub's suffix +semantics. `uploadKeyScope` (`pkg/business/datasource_file_filter.go`, beside +GitHub's own filter) is the one matcher. The connector applies it, not the +listing source: it is the `InScope` of an `objectstore.Scope` +(`pkg/datasource/objectstore/scope.go`), whose `Window` takes the store's +listing whole and keeps the matches, counting only them against +`max_objects`. So an excluded object is never fetched and never uses up the +cap, the closing listing names only matches, and replacing what lists the +prefix leaves the filter where it is. + +- **Refused at the call, never at sync.** `AddSource` and `UpdateSource` + compile every pattern and refuse an invalid one, more than 16, or one over 256 + characters as `INVALID_ARGUMENT` with a `google.rpc.BadRequest` violation and + a message that leads with the field path (`upload.include_patterns[1]: …` on + add, `include_patterns[1]: …` on update). The kit places a refusal under its + field by that leading path, because the TypeScript SDK carries no + error-detail types. +- **Narrowing deletes.** `UpdateSource` takes the patterns and file types for an + upload source (branch and paths are refused), under the same `expected` rule + as a GitHub edit, and queues the source's pull sync. That sync's closing + listing omits what the narrower filter excludes, and the store tombstones it + — unlike a GitHub snapshot, which only declares scope (below). +- **Access follows the source.** A source is (bucket, prefix, filter) → one + collection → one grant; two sources on one bucket with different filters are + two collections with different readers. The catalog description says so. +- **The filter is the host's whatever the store.** An S3-style listing takes a + prefix and a delimiter only, so the object-storage service needs nothing for + this. The source's `Scope` rides on its `objectstore.SourceConfig`, so the + envelope connector applies it to the service's listing before the version is + digested (#1211). + ## A datasource source is refreshed by three things, and one of them covers every provider Ranked by latency, a source's content is refreshed by a push (near-live), by a @@ -1304,3 +1341,86 @@ cause (v2 of that type; `datasourceAccessLostCodes` is the whole vocabulary a registry test holds the declaration to). Recovery is the ordinary one — a later snapshot clearing the degrade — which a repository made public again, or reconnected with a PAT or through the App, all reach. + +## Object storage is a client of the object-storage service, diffed against a listing the host retains + +The `upload` provider — "Object storage" to a person — reads a bucket under a +key prefix into a `collection` scope node. Its readers are source-scoped: a +bucket has no per-object access list, so the model to offer is one collection +per prefix, granted like any scope. `AddSource` refuses a webhook secret for it: +there is no push path (#1211 b). + +The host **re-uses the composed object-storage service** rather than keep a +store client of its own (the owner's direction, 2026-10-21). This module +composes it (`services/object-storage`, the `codefly.dev/object-storage` agent) +and accounts depends on its `grpc` endpoint; `work.go` dials it with the token +its agent publishes. The service takes a per-call bucket binding — kind, +endpoint, region, bucket, prefix and the source's access key, for that one call +— and owns the store protocol, every backend, the public-destination guard on +each dial and the size cap (codefly-dev/service-object-storage#108). The host +keeps what is per-source state: + +- **The sync reads through the connector.** `runUploadSync` takes a complete + snapshot through `objectstore.FilesConnector` (`Changes` from nothing), reads + it in batches with `FetchFiles`, delivers each object, and closes with the + listing the store tombstones the rest against. It still re-sends every object + in scope each sync; delivering only what the cursor says changed would change + what the consuming module receives, so it is its own change. +- **The retained listing is the cursor.** The connector versions a source by + the digest of its scoped listing under its binding. The host keeps those + listings in `datasource_object_manifests` (migration 59), tenant-owned — the + control plane has no grant — and pruned to the newest four per source by + `PutDatasourceObjectManifest`; a pruned or rebound version resyncs. Clause by + clause, in [CONFORMANCE.md](./code/pkg/datasource/connector/CONFORMANCE.md). +- **`objectstore.ServiceBucket` is the store.** It pages the listing (bounded; + past the bound it is `ErrListingIncomplete`, never a partial version) and + reads a batch in one `GetMany`, each object at `If-Match` its listed ETag. + The binding carries no prefix — the service would normalize it as a folder — + and every page filters on the source's literal key prefix (`/reports/` stays + `/reports/`), so an item keeps its full key; a listed key outside that prefix, + or a page past the service's page size, refuses the listing. Every call has a + deadline (30 s, five minutes for a batched read). + Before it lists or reads it asks `Capabilities` under the binding and refuses + a service that reports no bound caps or no batch size — a release from before + the binding would ignore it and serve its own bucket as the tenant's. +- **Conformant, and admitted.** A batch is read in one `GetMany` call (up to + the service's `max_get_many_items`, 1000), each object at its listed ETag, so + the provider passes the conformance suite through `ServiceBucket` over the + service's wire and is registered through `connector.Budgeted`: a sync spends + its credential's budget, keyed by a digest of the store's origin and access + key id derived from the source's configuration on every use (never stored, + so sources connected at any time share one quota on one key), and a + requested sync may spend the reserved share. Where a compatible service is composed, + `GetDatasourceCatalog` reports `accepts_new_sources` and `AddSource` takes a + new source with its prefix and key filter. That is the whole admin path a + composed consumer drives — catalog, `AddSource`, sync — with no portal step. +- **Admission needs a service that can read a bucket.** At start `work.go` + wires the composed service through `Service.SetObjectStorage`, which first + asks it, unbound, whether it reads a batch (`objectstore.CheckService`) and + refuses to wire one that does not. A composition without the service, or with one + from before the binding (0.0.18 and earlier; this module pins 0.0.19), + leaves object storage unavailable: the catalog says so, `AddSource` refuses + with `ErrObjectStorageUnavailable`, and an existing source's sync fails by + name. +- **A sync never empties a source that has files** — a host-owned invariant, + whatever any service version answers. A closing snapshot with no files + tombstones every entry mirrored from the source, and a service that wrongly + answers "empty" (a rewritten scope, a discarded truncation) has happened. + So a listing that comes back empty for a source whose last closing snapshot + named objects — `datasource_sources.last_snapshot_items`, migration 60, + written only when a snapshot is sent; unknown on a source synced before it, + which counts as having files — is refused with + `ErrUploadEmptyListingRefused` (job code + `datasource.upload_empty_listing_refused`), closes nothing, and leaves the + collection as it is. Emptying a source is an explicit operation. +- **Each refusal gets the action its reason names**, not one generic retry: + a throttle (`STORE_THROTTLED`) is retried no earlier than the store's reset; + an object over the service's cap (`OBJECT_TOO_LARGE`) is skipped and the + sync goes on, then surfaces it without a closing snapshot; a guard refusal + (`DESTINATION_REFUSED`) or a rejected access key (`CREDENTIAL_REJECTED`) + parks the source as degraded, with a fixed reason asking for a new endpoint + or a valid key, and fails the job terminally — unless the park itself cannot + be written, when the job retries instead, so a source never stops with + nothing on it to say why (`parkDatasourceSource`, the park the GitHub + unreadable-public-repository path goes through as well); an incomplete, too-long or too-large + listing is the terminal incomplete-listing disposition. diff --git a/module/services/accounts/code/external/saas-starter/object-storage/saas-starter_object-storage_grpc.pb.go b/module/services/accounts/code/external/saas-starter/object-storage/saas-starter_object-storage_grpc.pb.go new file mode 100644 index 000000000..fca2bb91d --- /dev/null +++ b/module/services/accounts/code/external/saas-starter/object-storage/saas-starter_object-storage_grpc.pb.go @@ -0,0 +1,60 @@ +// Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.36.11 +// protoc (unknown) +// source: saas-starter_object-storage_grpc.proto + +package object_storage + +import ( + reflect "reflect" + unsafe "unsafe" + + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" +) + +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) + +var File_saas_starter_object_storage_grpc_proto protoreflect.FileDescriptor + +const file_saas_starter_object_storage_grpc_proto_rawDesc = "" + + "\n" + + "&saas-starter_object-storage_grpc.protoBqB!SaasStarterObjectStorageGrpcProtoP\x01ZJgithub.com/codefly-dev/cli/pkg/builder/clients/saas-starter/object-storage" + +var file_saas_starter_object_storage_grpc_proto_goTypes = []any{} +var file_saas_starter_object_storage_grpc_proto_depIdxs = []int32{ + 0, // [0:0] is the sub-list for method output_type + 0, // [0:0] is the sub-list for method input_type + 0, // [0:0] is the sub-list for extension type_name + 0, // [0:0] is the sub-list for extension extendee + 0, // [0:0] is the sub-list for field type_name +} + +func init() { file_saas_starter_object_storage_grpc_proto_init() } +func file_saas_starter_object_storage_grpc_proto_init() { + if File_saas_starter_object_storage_grpc_proto != nil { + return + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: unsafe.Slice(unsafe.StringData(file_saas_starter_object_storage_grpc_proto_rawDesc), len(file_saas_starter_object_storage_grpc_proto_rawDesc)), + NumEnums: 0, + NumMessages: 0, + NumExtensions: 0, + NumServices: 0, + }, + GoTypes: file_saas_starter_object_storage_grpc_proto_goTypes, + DependencyIndexes: file_saas_starter_object_storage_grpc_proto_depIdxs, + }.Build() + File_saas_starter_object_storage_grpc_proto = out.File + file_saas_starter_object_storage_grpc_proto_goTypes = nil + file_saas_starter_object_storage_grpc_proto_depIdxs = nil +} diff --git a/module/services/accounts/code/go.mod b/module/services/accounts/code/go.mod index 9ce81b4de..d320900bf 100644 --- a/module/services/accounts/code/go.mod +++ b/module/services/accounts/code/go.mod @@ -18,6 +18,7 @@ require ( github.com/codefly-dev/module-saas-starter/module/frontendpayload v0.0.0-20261021192505-6fd56dae2e04 github.com/codefly-dev/sdk-go v0.4.1 github.com/codefly-dev/sdk-go/workcontext v0.1.1-0.20261018052022-70dc7a546458 + github.com/codefly-dev/service-object-storage v0.0.19 github.com/codefly-dev/service-postgres/libs/go v0.0.0-20260929200903-bae6c31c7211 github.com/codefly-dev/service-redis/cache v0.3.0 github.com/go-webauthn/webauthn v0.18.0 diff --git a/module/services/accounts/code/go.sum b/module/services/accounts/code/go.sum index d5ecc2ab7..9f99cf9b9 100644 --- a/module/services/accounts/code/go.sum +++ b/module/services/accounts/code/go.sum @@ -82,6 +82,8 @@ github.com/codefly-dev/sdk-go v0.4.1 h1:WUKYWxXfojTr88SkLtq0T8aqpV0o5pR/muUUcsUU github.com/codefly-dev/sdk-go v0.4.1/go.mod h1:zr9ts3kkWNv2XEFUXZB+ydfW6x5b6/NbUvBm4KVYNqs= github.com/codefly-dev/sdk-go/workcontext v0.1.1-0.20261018052022-70dc7a546458 h1:wBOL7QsuoC0HsS43gsH2+tpsipk4os077C2VUDkMUHg= github.com/codefly-dev/sdk-go/workcontext v0.1.1-0.20261018052022-70dc7a546458/go.mod h1:hwtrdGtgh8i2xzjgihyd4wY2u+UK7nHQz30OGKL0n+M= +github.com/codefly-dev/service-object-storage v0.0.19 h1:PwrpBjJZ+lKWpuz1eKEBVAQsRkUQtd8Jj8Krfbb/oa8= +github.com/codefly-dev/service-object-storage v0.0.19/go.mod h1:EgEr/+b15Cv8Mq54ONfILBuWnBcAk8t62CbXISbRONU= github.com/codefly-dev/service-postgres/libs/go v0.0.0-20260929200903-bae6c31c7211 h1:4KAPx8haCA/h4Kx5mB7D0rMriwLLKiOW9G9mMWuiQDw= github.com/codefly-dev/service-postgres/libs/go v0.0.0-20260929200903-bae6c31c7211/go.mod h1:5mwz8oM8INcukJLc6C84zdRnmDQ/gkMNI8GxWMwgk+8= github.com/codefly-dev/service-redis/cache v0.3.0 h1:F0mb7T1xHJBJDgEmI3kPriyb2EmPxKgtmOxSIUylX7s= @@ -93,8 +95,8 @@ github.com/containerd/errdefs/pkg v0.3.0/go.mod h1:NJw6s9HwNuRhnjJhM7pylWwMyAkmC github.com/cyphar/filepath-securejoin v0.7.0 h1:s0Y3ITPy6sQn5xt54DuYvTF8hu134ooYLUb58DX/HjE= github.com/cyphar/filepath-securejoin v0.7.0/go.mod h1:ymLGms/u3BYaviIiuKFnUx8EkQEZeK6cInNoAPJA3o4= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= -github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= +github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk= github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E= github.com/dlclark/regexp2 v1.12.0 h1:0j4c5qQmnC6XOWNjP3PIXURXN2gWx76rd3KvgdPkCz8= @@ -227,8 +229,8 @@ github.com/kevinburke/ssh_config v1.6.0 h1:J1FBfmuVosPHf5GRdltRLhPJtJpTlMdKTBjRg github.com/kevinburke/ssh_config v1.6.0/go.mod h1:q2RIzfka+BXARoNexmF9gkxEX7DmvbW9P4hIVx2Kg4M= github.com/keybase/go-keychain v0.0.1 h1:way+bWYa6lDppZoZcgMbYsvC7GxljxrskdNInRtuthU= github.com/keybase/go-keychain v0.0.1/go.mod h1:PdEILRW3i9D8JcdM+FmY6RwkHGnhHxXwkPPMeUgOK1k= -github.com/klauspost/cpuid/v2 v2.2.10 h1:tBs3QSyvjDyFTq3uoc/9xFpCuOsJQFNPiAhYdw2skhE= -github.com/klauspost/cpuid/v2 v2.2.10/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0= +github.com/klauspost/cpuid/v2 v2.4.0 h1:S6Hrbc7+ywsr0r+RLapfGBHfyefhCTwEh3A0tV913Dw= +github.com/klauspost/cpuid/v2 v2.4.0/go.mod h1:19jmZ9mjzoF//ddRSUsv0zfBTJWh3QJh9FNxZTMrGxU= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= diff --git a/module/services/accounts/code/pkg/adapters/datasource_handler.go b/module/services/accounts/code/pkg/adapters/datasource_handler.go index 8acdbf95f..0c711b005 100644 --- a/module/services/accounts/code/pkg/adapters/datasource_handler.go +++ b/module/services/accounts/code/pkg/adapters/datasource_handler.go @@ -107,6 +107,10 @@ func (h *datasourceConnectHandler) AddSource( Prefix: up.Prefix, AccessKeyID: up.AccessKeyId, MaxObjects: int(up.MaxObjects), + + IncludePatterns: up.IncludePatterns, + ExcludePatterns: up.ExcludePatterns, + FileExtensions: up.FileExtensions, } } source, err := h.svc.AddSource(ctx, actorID, input) @@ -134,6 +138,9 @@ func (h *datasourceConnectHandler) UpdateSource( Branch: req.Msg.Branch, Paths: req.Msg.Paths, FileExtensions: req.Msg.FileExtensions, + + IncludePatterns: req.Msg.IncludePatterns, + ExcludePatterns: req.Msg.ExcludePatterns, } // The scope the caller was shown. The descriptor marks it required, so an // absent one is already refused before this handler runs; it is mapped @@ -144,6 +151,9 @@ func (h *datasourceConnectHandler) UpdateSource( Branch: expected.GetBranch(), Paths: expected.GetPaths(), FileExtensions: expected.GetFileExtensions(), + + IncludePatterns: expected.GetIncludePatterns(), + ExcludePatterns: expected.GetExcludePatterns(), } } source, jobID, err := h.svc.UpdateSource(ctx, actorID, input) @@ -436,6 +446,10 @@ func datasourceSourceToProto(source *business.DatasourceSource, registry *connec Prefix: source.Upload.Prefix, AccessKeyId: source.Upload.AccessKeyID, MaxObjects: uint32(source.Upload.MaxObjects), + + IncludePatterns: source.Upload.IncludePatterns, + ExcludePatterns: source.Upload.ExcludePatterns, + FileExtensions: source.Upload.FileExtensions, } } if source.LastSyncedAt != nil { diff --git a/module/services/accounts/code/pkg/adapters/datasource_handler_test.go b/module/services/accounts/code/pkg/adapters/datasource_handler_test.go index afbf185fb..7ac793a5f 100644 --- a/module/services/accounts/code/pkg/adapters/datasource_handler_test.go +++ b/module/services/accounts/code/pkg/adapters/datasource_handler_test.go @@ -5,6 +5,7 @@ import ( "time" "accounts/pkg/business" + "accounts/pkg/datasource/objectstore" gen "accounts/pkg/gen/saas/accounts/v1" ) @@ -36,6 +37,12 @@ func TestAPICredentialKindRoundTrip(t *testing.T) { func TestDatasourceCatalog_ProjectsTheRegistry(t *testing.T) { svc, _ := business.NewService(nil) svc.SetDatasourceConnector(nil, nil, "") + for _, p := range datasourceCatalog(svc.DatasourceCatalog()).GetProviders() { + if p.GetConnector() == business.DatasourceProviderUpload && p.GetAcceptsNewSources() { + t.Fatal("object storage accepts new sources with no object-storage service composed") + } + } + svc.SetDatasourceUploadBucketFactory(func(objectstore.SourceConfig, string, string) objectstore.Bucket { return nil }) catalog := datasourceCatalog(svc.DatasourceCatalog()) byKey := map[string]*gen.DatasourceProviderDescriptor{} for _, p := range catalog.GetProviders() { @@ -53,13 +60,24 @@ func TestDatasourceCatalog_ProjectsTheRegistry(t *testing.T) { len(github.GetConfigFields()) != 4 || github.GetConfigFields()[0].GetKey() != "repo" || !github.GetSupportsWebhook() { t.Fatalf("GitHub entry = %v", github) } - for _, key := range []string{business.DatasourceProviderAPI, business.DatasourceProviderCrawler, business.DatasourceProviderUpload} { + upload := byKey[business.DatasourceProviderUpload] + if !upload.GetConformant() || !upload.GetAcceptsNewSources() || upload.GetConformanceGap() != "" || + upload.GetInterface() != gen.DatasourceConnectorInterface_DATASOURCE_CONNECTOR_INTERFACE_FILES || + upload.GetBudget().GetMaxItemsPerCall() != 1000 || upload.GetBudget().GetMaxItemBytes() != 5<<20 { + t.Fatalf("object storage entry = %v, want conformant, accepting new sources, budgeted", upload) + } + for _, key := range []string{business.DatasourceProviderAPI, business.DatasourceProviderCrawler} { p := byKey[key] if p == nil || p.GetConformant() || p.GetAcceptsNewSources() || p.GetConformanceGap() == "" || p.GetBudget() != nil || p.GetProvider() == gen.DatasourceProvider_DATASOURCE_PROVIDER_UNSPECIFIED { t.Fatalf("%s entry = %v, want listed, non-conformant, with a gap, accepting no new source", key, p) } } + // A bucket has no per-object access list; the catalog says so before a + // person connects one. + if r := byKey[business.DatasourceProviderUpload].GetReadersModel(); r != gen.DatasourceReadersModel_DATASOURCE_READERS_MODEL_SOURCE_SCOPED { + t.Fatalf("object storage readers model = %v, want source-scoped", r) + } if datasourceCatalog(nil).GetProviders() != nil { t.Fatal("an unconfigured connector offers nothing") } diff --git a/module/services/accounts/code/pkg/auth/pg/session_store_test.go b/module/services/accounts/code/pkg/auth/pg/session_store_test.go index ba1d8e044..ad5ba5bfc 100644 --- a/module/services/accounts/code/pkg/auth/pg/session_store_test.go +++ b/module/services/accounts/code/pkg/auth/pg/session_store_test.go @@ -63,7 +63,7 @@ func runSessionStoreTests(m *testing.M) int { setupDone := testdb.MeasureSetup("auth-db", []string{"store"}, 120*time.Second) deps, err := sdk.WithDependencies(ctx, sdk.WithDebug(), - sdk.WithExcludedDependencies("cache", "vault"), + sdk.WithExcludedDependencies("object-storage", "cache", "vault"), sdk.WithNamingScope("pgauth-test"), sdk.WithTimeout(120*time.Second), sdk.WithSilence("store"), diff --git a/module/services/accounts/code/pkg/billing/pg/store_test.go b/module/services/accounts/code/pkg/billing/pg/store_test.go index d672d63a8..f46f361ee 100644 --- a/module/services/accounts/code/pkg/billing/pg/store_test.go +++ b/module/services/accounts/code/pkg/billing/pg/store_test.go @@ -44,7 +44,7 @@ func runBillingStoreTests(m *testing.M) int { setupDone := testdb.MeasureSetup("billing-db", []string{"store"}, 120*time.Second) deps, err := sdk.WithDependencies(ctx, sdk.WithDebug(), - sdk.WithExcludedDependencies("cache", "vault"), + sdk.WithExcludedDependencies("object-storage", "cache", "vault"), sdk.WithNamingScope("pgbilling-test"), sdk.WithTimeout(120*time.Second), sdk.WithSilence("store"), diff --git a/module/services/accounts/code/pkg/business/audit_registry.go b/module/services/accounts/code/pkg/business/audit_registry.go index 1c03099d9..a6d047c27 100644 --- a/module/services/accounts/code/pkg/business/audit_registry.go +++ b/module/services/accounts/code/pkg/business/audit_registry.go @@ -884,8 +884,9 @@ var auditEventCatalog = []AuditEventDefinition{ // reads this declaration for. An edit never moves the boundary, but it is // the collection whose contents the next snapshot replaces, so whoever may // read that collection is exactly who needs to see it. - mutation(EventDatasourceSourceUpdated, CategorySystem, "A connected datasource's branch or path filters were changed.", - str("repo"), str("branch"), str("previous_branch"), strs("paths"), strs("file_extensions"), str("boundary")), + mutation(EventDatasourceSourceUpdated, CategorySystem, "A connected datasource's branch, path or key filters were changed.", + str("repo"), str("branch"), str("previous_branch"), strs("paths"), strs("file_extensions"), str("boundary"), + str("bucket"), str("prefix"), strs("include_patterns"), strs("exclude_patterns")), observation(EventDatasourceSyncCompleted, CategorySystem, "A datasource ingestion job completed.", sourceSyncFields...), observation(EventDatasourceSyncFailed, CategorySystem, "A datasource ingestion attempt failed and may retry.", sourceSyncFields...), observation(EventDatasourceChangeSetCompiled, CategorySystem, "A GitHub delivery was compiled into a change set.", diff --git a/module/services/accounts/code/pkg/business/connector_conformance_record_test.go b/module/services/accounts/code/pkg/business/connector_conformance_record_test.go index 1f272fa20..850b5589d 100644 --- a/module/services/accounts/code/pkg/business/connector_conformance_record_test.go +++ b/module/services/accounts/code/pkg/business/connector_conformance_record_test.go @@ -67,7 +67,7 @@ var conformanceRecordProviders = map[string]string{ DatasourceProviderGitHub: "", DatasourceProviderAPI: gapAPIDatasource, DatasourceProviderCrawler: gapCrawlerDatasource, - DatasourceProviderUpload: gapUploadDatasource, + DatasourceProviderUpload: "", } func TestConformanceRecordCoversEveryRegisteredProvider(t *testing.T) { diff --git a/module/services/accounts/code/pkg/business/datasource_budget.go b/module/services/accounts/code/pkg/business/datasource_budget.go index 8ededbb28..1d5f1d1e5 100644 --- a/module/services/accounts/code/pkg/business/datasource_budget.go +++ b/module/services/accounts/code/pkg/business/datasource_budget.go @@ -2,6 +2,11 @@ package business import ( "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "net/url" + "strings" "time" "accounts/pkg/datasource/connector" @@ -75,6 +80,24 @@ func (d datasourceScheduler) Blocked(ctx context.Context, src connector.Source, }) } +// uploadCredentialKey names the credential an object-storage source reads with: +// its access key at its store. It is derived from the source's own +// configuration on every use — never stored at connect — so every source on +// one key spends one quota and honours one provider block, whenever it was +// connected and whatever the host was configured with then. The identity is +// the store's origin and the access key id, both plain configuration on the +// same row (the secret key is not part of it), so a digest of them stores no +// fingerprint of anything secret. +func uploadCredentialKey(cfg *UploadDatasourceConfig) string { + scheme, host := "", cfg.Endpoint + if origin, err := url.Parse(cfg.Endpoint); err == nil && origin.Host != "" { + scheme, host = strings.ToLower(origin.Scheme), strings.ToLower(origin.Host) + } + identity, _ := json.Marshal([]string{scheme, host, cfg.AccessKeyID}) + sum := sha256.Sum256(append([]byte("datasource-upload-budget\x00"), identity...)) + return "upload:credential:" + hex.EncodeToString(sum[:]) +} + // datasourceCredentialKey names the credential a source's operations spend: // an App installation is shared by every source it covers, a pasted token is // the source's own, and an unauthenticated public read is metered for the @@ -83,6 +106,8 @@ func datasourceCredentialKey(source *DatasourceSource) string { switch { case source.Provider == DatasourceProviderAPI && source.API != nil && source.API.CredentialBudgetKey != "": return source.API.CredentialBudgetKey + case source.Provider == DatasourceProviderUpload && source.Upload != nil: + return uploadCredentialKey(source.Upload) case source.Provider != DatasourceProviderGitHub: return source.Provider + ":source:" + source.ID case source.GitHubInstallationID != "": diff --git a/module/services/accounts/code/pkg/business/datasource_connectors.go b/module/services/accounts/code/pkg/business/datasource_connectors.go index 2ba1db5fc..3844e601f 100644 --- a/module/services/accounts/code/pkg/business/datasource_connectors.go +++ b/module/services/accounts/code/pkg/business/datasource_connectors.go @@ -7,18 +7,21 @@ import ( "accounts/pkg/datasource/connector" "accounts/pkg/datasource/github" + "accounts/pkg/datasource/objectstore" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" ) // The host's datasource connectors, on the envelope in pkg/datasource/connector. -// GitHub is the one conformant connector: it passes the conformance suite -// (pkg/datasource/github/connector_conformance_test.go). The generic API, the -// crawler and object storage still run on their own engines, outside the -// envelope; they are registered non-conformant with their gap, so their -// existing sources keep running and new ones are refused until each passes the -// suite (the owner's settled question 7). +// Two are conformant, each passing the conformance suite beside it: GitHub +// (pkg/datasource/github/connector_conformance_test.go) and object storage, +// read through the composed object-storage service +// (pkg/datasource/objectstore/connector_conformance_test.go). The generic API +// and the crawler still run on their own engines, outside the envelope; they +// are registered non-conformant with their gap, so their existing sources keep +// running and new ones are refused until each passes the suite (the owner's +// settled question 7). // Why each legacy provider is not on the envelope yet, per connector. What // remains for each, clause by clause, and what it already honours, is in @@ -33,14 +36,6 @@ import ( const ( gapAPIDatasource = "one untyped response with no item identity, no cursor and no deletions; to be rebuilt as a records connector or retired" gapCrawlerDatasource = "no cursor and no deletions (a page leaving the sitemap is never removed), and one fetch per page" - // Object storage DOES report deletions: runUploadSync closes a sync that saw - // the whole folder with the complete listing of what it delivered, which the - // store tombstones the rest against. This sentence used to say the opposite - // — "a removed object is never removed" — which told every tenant of an - // object-storage source that their deletions were being ignored, when the - // mirror listing is exactly the mechanism that applies them. What remains is - // the cursor: with no version to diff from, every sync re-sends every object. - gapUploadDatasource = "no cursor, so every sync re-sends every object, and one fetch per object; deletions ARE applied, through the complete listing that closes a whole-folder sync" ) // newDatasourceConnectorRegistry builds the host's registry. A registration @@ -69,13 +64,10 @@ func (s *Service) newDatasourceConnectorRegistry() *connector.Registry { CredentialModes: []connector.CredentialMode{connector.CredentialNone}, Gap: gapCrawlerDatasource, })) - must(r.RegisterNonConformant(connector.Descriptor{ - Key: DatasourceProviderUpload, DisplayName: "Object storage", - Description: "An S3-compatible bucket; objects under a prefix are pulled on sync.", - Interface: connector.InterfaceFiles, - CredentialModes: []connector.CredentialMode{connector.CredentialStaticSecret}, - Gap: gapUploadDatasource, - })) + // Object storage reads through the composed object-storage service and + // passes the suite over its wire (objectstore/connector_conformance_test.go); + // metered like GitHub, per source credential. + must(r.Register(connector.Budgeted(objectstore.NewFilesConnector(s.uploadBucketForSource, objectManifests{s.store}), datasourceScheduler{s: s}))) return r } @@ -117,7 +109,7 @@ func (s *Service) DatasourceCatalog() []DatasourceCatalogEntry { AcceptsOperations: d.Key == DatasourceProviderAPI, OperationsGap: gap, OAuth: s.datasourceOAuth[d.Key], - AcceptsNewSources: s.datasourceConnectors.AdmitNewSource(d.Key) == nil, + AcceptsNewSources: s.admitNewDatasource(d.Key) == nil, LiveDeliveryConfigured: s.DatasourceLiveDeliveryConfigured(d.Key), }) } @@ -138,14 +130,25 @@ func DatasourceConformance(registry *connector.Registry, provider string) (bool, return d.Conformant, d.Gap } +// ErrObjectStorageUnavailable: no composed object-storage service here can read +// a tenant's bucket, so an object-storage source is neither admitted nor synced. +var ErrObjectStorageUnavailable = errors.New("object storage is not available here: no composed object-storage service serves a per-call binding with a batched read") + // admitNewDatasource refuses a new source of a provider the registry does not -// admit, before anything is validated, sealed or stored. +// admit, or that nothing composed here can serve, before anything is +// validated, sealed or stored. The catalog reports the same answer. func (s *Service) admitNewDatasource(provider string) error { if s.datasourceConnectors == nil { return status.Error(codes.FailedPrecondition, "datasource connector is not configured") } err := s.datasourceConnectors.AdmitNewSource(provider) switch { + case err == nil && provider == DatasourceProviderUpload && s.newUploadBucket == nil: + // Conformant, but nothing here can read a bucket: no object-storage + // service is composed, or the one composed predates the per-call + // binding and the batched read (work.go asks it at start). A source + // admitted now would fail every sync. + return status.Error(codes.FailedPrecondition, ErrObjectStorageUnavailable.Error()) case err == nil: return nil case errors.Is(err, connector.ErrUnknownConnector): @@ -159,12 +162,59 @@ func (s *Service) admitNewDatasource(provider string) error { // tenancy, and — for GitHub — its repository, branch and the host's own scope. func connectorSource(source *DatasourceSource) connector.Source { src := connector.Source{ID: source.ID, OrgID: source.OrgID, BoundaryNodeID: source.BoundaryNodeID, CredentialKey: datasourceCredentialKey(source), PersonalOwnerUserID: source.PersonalOwnerUserID} - if source.Provider == DatasourceProviderGitHub { + switch { + case source.Provider == DatasourceProviderGitHub: src.Config = github.SourceConfig{Repo: source.Repo, Branch: source.Branch, InScope: datasourceFileScope(source)} + case source.Provider == DatasourceProviderUpload && source.Upload != nil: + cfg := source.Upload + src.Config = objectstore.SourceConfig{ + Endpoint: cfg.Endpoint, Region: cfg.Region, Bucket: cfg.Bucket, Prefix: cfg.Prefix, + Scope: objectstore.Scope{InScope: uploadKeyScope(cfg), MaxObjects: cfg.MaxObjects}, + } } return src } +// uploadBucketForSource resolves one object-storage source's Bucket, per call: +// the host decrypts the source's secret access key and binds it to that call, +// and the connector never sees it. +func (s *Service) uploadBucketForSource(ctx context.Context, src connector.Source) (objectstore.Bucket, error) { + if s.newUploadBucket == nil { + return nil, ErrObjectStorageUnavailable + } + source, err := s.store.GetDatasourceSourceByID(ctx, src.ID) + if err != nil { + return nil, err + } + if source == nil || source.OrgID != src.OrgID || source.Provider != DatasourceProviderUpload || source.Upload == nil { + return nil, fmt.Errorf("datasource source %s is not an object-storage source of org %s", src.ID, src.OrgID) + } + secret, err := s.datasourceCipher.DecryptSecret(ctx, DatasourceConnectorSecretPurpose(source.ID), source.CredentialSecretRef) + if err != nil { + return nil, fmt.Errorf("decrypt secret access key: %w", err) + } + cfg, _ := connectorSource(source).Config.(objectstore.SourceConfig) + return s.newUploadBucket(cfg, source.Upload.AccessKeyID, secret), nil +} + +// objectManifests retains an object-storage source's listings in the store, +// inside the source's org transaction. +type objectManifests struct{ store Store } + +func (m objectManifests) Put(ctx context.Context, src connector.Source, version string, items []objectstore.Item) error { + return m.store.WithOrgTx(ctx, src.OrgID, func(ctx context.Context) error { + return m.store.PutDatasourceObjectManifest(ctx, src.OrgID, src.ID, version, items) + }) +} + +func (m objectManifests) Get(ctx context.Context, src connector.Source, version string) (items []objectstore.Item, ok bool, err error) { + err = m.store.WithOrgTx(ctx, src.OrgID, func(ctx context.Context) error { + items, ok, err = m.store.GetDatasourceObjectManifest(ctx, src.OrgID, src.ID, version) + return err + }) + return items, ok, err +} + // githubRemoteForSource resolves the credentialed GitHub client for one source, // per call: the host holds the credential and the connector never sees it. func (s *Service) githubRemoteForSource(ctx context.Context, src connector.Source) (github.Remote, error) { diff --git a/module/services/accounts/code/pkg/business/datasource_connectors_test.go b/module/services/accounts/code/pkg/business/datasource_connectors_test.go index 7327aa217..269ba244b 100644 --- a/module/services/accounts/code/pkg/business/datasource_connectors_test.go +++ b/module/services/accounts/code/pkg/business/datasource_connectors_test.go @@ -1,13 +1,20 @@ package business_test import ( + "context" "os" "path/filepath" + "reflect" "regexp" + "strings" "testing" "accounts/pkg/business" "accounts/pkg/datasource/connector" + "accounts/pkg/datasource/objectstore" + + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" ) // suiteCall is how a connector's own tests run the conformance suite. @@ -15,18 +22,36 @@ var suiteCall = regexp.MustCompile(`connectortest\.Run[A-Z][A-Za-z]*\(t,`) // TestEveryConformantConnectorRunsTheSuite is the conformance gate, the same // shape as the boundary gate: a connector the host registers as conformant must -// have a conformance test beside it, in pkg/datasource//, that runs the -// shared suite. Registering one as conformant without that test is a red build. +// have a conformance test beside it, in the package that implements it, that +// runs the shared suite. Registering one as conformant without that test is a +// red build. The package is the implementation's, found through the meter that +// wraps it, not the provider key's: object storage is keyed "upload" on every +// stored source and implemented in pkg/datasource/objectstore. func TestEveryConformantConnectorRunsTheSuite(t *testing.T) { svc, _ := business.NewService(nil) svc.SetDatasourceConnector(nil, nil, "") conformant := svc.DatasourceConnectors().Conformant() - if len(conformant) == 0 { - t.Fatal("the host registers no conformant connector") + if len(conformant) < 2 { + t.Fatalf("the host registers %d conformant connectors, want GitHub and object storage", len(conformant)) } for _, c := range conformant { key := c.Descriptor().Key - dir := filepath.Join("..", "datasource", key) + impl := any(c) + for { + wrapped, ok := impl.(interface { + Unwrap() connector.FilesConnector + }) + if !ok { + break + } + impl = wrapped.Unwrap() + } + pkg := reflect.TypeOf(impl).Elem().PkgPath() + rel, ok := strings.CutPrefix(pkg, "accounts/pkg/") + if !ok { + t.Fatalf("connector %q is implemented outside accounts/pkg (%s)", key, pkg) + } + dir := filepath.Join("..", rel) matches, err := filepath.Glob(filepath.Join(dir, "*_test.go")) if err != nil { t.Fatal(err) @@ -50,12 +75,13 @@ func TestEveryConformantConnectorRunsTheSuite(t *testing.T) { // TestLegacyProvidersAreRegisteredWithTheirGap holds the registry to the owner's // settled rule for providers still off the envelope: registered, flagged -// non-conformant with a named gap, not offered, not admitted. +// non-conformant with a named gap, not offered, not admitted — and the two on +// it, GitHub and object storage, conformant, source-scoped and admitted. func TestLegacyProvidersAreRegisteredWithTheirGap(t *testing.T) { svc, _ := business.NewService(nil) svc.SetDatasourceConnector(nil, nil, "") reg := svc.DatasourceConnectors() - for _, key := range []string{business.DatasourceProviderAPI, business.DatasourceProviderCrawler, business.DatasourceProviderUpload} { + for _, key := range []string{business.DatasourceProviderAPI, business.DatasourceProviderCrawler} { d, ok := reg.Descriptor(key) if !ok || d.Conformant || d.Gap == "" { t.Fatalf("%s: descriptor %+v, want registered non-conformant with a gap", key, d) @@ -64,12 +90,53 @@ func TestLegacyProvidersAreRegisteredWithTheirGap(t *testing.T) { t.Fatalf("%s: admission = %v, want ErrNonConformant", key, err) } } + // Object storage is admitted only where a compatible object-storage + // service is composed (TestObjectStorageIsAdmittedOnlyWhereAServiceCanReadIt). + svc.SetDatasourceUploadBucketFactory(func(objectstore.SourceConfig, string, string) objectstore.Bucket { return nil }) for _, e := range svc.DatasourceCatalog() { if e.AcceptsNewSources != (e.Descriptor.Conformant && e.Descriptor.Readers != connector.ReadersTranslated) { t.Fatalf("the catalog says %s accepts new sources = %v", e.Descriptor.Key, e.AcceptsNewSources) } } - if d, ok := reg.Descriptor(business.DatasourceProviderGitHub); !ok || !d.Conformant || d.Readers != connector.ReadersSourceScoped { - t.Fatalf("github: %+v, want conformant and source-scoped", d) + for _, key := range []string{business.DatasourceProviderGitHub, business.DatasourceProviderUpload} { + if d, ok := reg.Descriptor(key); !ok || !d.Conformant || d.Readers != connector.ReadersSourceScoped { + t.Fatalf("%s: %+v, want conformant and source-scoped", key, d) + } + if err := reg.AdmitNewSource(key); err != nil { + t.Fatalf("%s: admission = %v, want admitted", key, err) + } + } +} + +// TestObjectStorageIsAdmittedOnlyWhereAServiceCanReadIt: object storage is +// conformant, but a host with no composed object-storage service — or one +// whose service predates the per-call binding, which work.go leaves unwired — +// cannot sync a bucket. The catalog says so, and AddSource refuses by name +// before anything is sealed or stored. +func TestObjectStorageIsAdmittedOnlyWhereAServiceCanReadIt(t *testing.T) { + svc, _ := business.NewService(nil) + svc.SetDatasourceConnector(nil, nil, "") + accepts := func() bool { + for _, e := range svc.DatasourceCatalog() { + if e.Descriptor.Key == business.DatasourceProviderUpload { + return e.AcceptsNewSources + } + } + t.Fatal("object storage is not in the catalog") + return false + } + if accepts() { + t.Fatal("the catalog accepts object-storage sources with no object-storage service composed") + } + _, err := svc.AddSource(context.Background(), "actor-1", business.AddSourceInput{ + OrgID: "11111111-1111-1111-1111-111111111111", Provider: business.DatasourceProviderUpload, CollectionLabel: "guides", Credential: "secretkey", + Upload: &business.UploadDatasourceConfig{Endpoint: "https://store.example.com", Region: "us-east-1", Bucket: "docs", AccessKeyID: "AKIAEXAMPLE"}, + }) + if status.Code(err) != codes.FailedPrecondition || !strings.Contains(err.Error(), business.ErrObjectStorageUnavailable.Error()) { + t.Fatalf("AddSource(upload) with no object storage = %v, want the unavailable refusal", err) + } + svc.SetDatasourceUploadBucketFactory(func(objectstore.SourceConfig, string, string) objectstore.Bucket { return nil }) + if !accepts() { + t.Fatal("the catalog refuses object-storage sources with an object-storage service composed") } } diff --git a/module/services/accounts/code/pkg/business/datasource_deliveries.go b/module/services/accounts/code/pkg/business/datasource_deliveries.go index 9145e80a5..b5f46bd88 100644 --- a/module/services/accounts/code/pkg/business/datasource_deliveries.go +++ b/module/services/accounts/code/pkg/business/datasource_deliveries.go @@ -356,22 +356,16 @@ func (s *Service) parkUnreadablePublicSource(ctx context.Context, source *Dataso w := wool.Get(ctx).In("parkUnreadablePublicSource") terminal := jobs.NewProcessingError("datasource.public_repository_unreadable", DatasourceReasonPublicRepositoryUnreadable+" This sync job will not retry.", false) - // Already parked for this: the job still stops, but the row is not rewritten - // and no second access_lost is recorded. A source is parked out of the - // reconcile sweep, so repeats arrive from "Sync now", which ignores status. - if source.Status == DatasourceStatusDegraded && source.StatusReason == DatasourceReasonPublicRepositoryUnreadable { - return terminal - } - if err := s.store.WithControlPlane(ctx, func(ctx context.Context) error { - return s.store.MarkDatasourceSourceDegraded(ctx, source.ID, PublicRepositoryUnreadableDegradeReason()) - }); err != nil { - // The park is what makes this visible; without it the tenant would see a - // source that simply stopped. Report the original failure so the job - // retries and the park is attempted again, rather than going terminal - // having recorded nothing. + parked, err := s.parkDatasourceSource(ctx, source, PublicRepositoryUnreadableDegradeReason()) + if err != nil { w.Warn("park unreadable public source failed", wool.Field("source", source.ID), wool.ErrField(err)) return cause } + // Already parked for this: the job still stops, and no second access_lost + // is recorded. + if !parked { + return terminal + } s.emit(ctx, source.ID, "system", EventDatasourceSourceAccessLost, "datasource", source.ID, source.OrgID, map[string]any{ "repo": source.Repo, @@ -384,6 +378,26 @@ func (s *Service) parkUnreadablePublicSource(ctx context.Context, source *Dataso return terminal } +// parkDatasourceSource degrades a source a retry cannot fix, with the reason a +// person acts on, so the reconcile sweep stops retrying it. It reports false, +// writing nothing, when the source already stands parked for that reason: a +// parked source is out of the sweep, so repeats arrive from "Sync now", which +// ignores status. On an error the caller reports a RETRYABLE failure, never its +// terminal one: the park is what makes the stop visible, and a job that went +// terminal without it would leave a source that simply stopped, with nothing +// on it to say why. +func (s *Service) parkDatasourceSource(ctx context.Context, source *DatasourceSource, reason DatasourceDegradeReason) (bool, error) { + if source.Status == DatasourceStatusDegraded && source.StatusReason == reason.String() { + return false, nil + } + if err := s.store.WithControlPlane(ctx, func(ctx context.Context) error { + return s.store.MarkDatasourceSourceDegraded(ctx, source.ID, reason) + }); err != nil { + return false, err + } + return true, nil +} + // CompileGitHubDelivery turns one raw push delivery into a change set, through // the GitHub connector. A delivery is a trigger, not a diff: after the branch // filter, the compiler asks the connector for the changes from the source's @@ -1032,7 +1046,8 @@ func forcedReconcileMode(mode string) bool { // connecting is an explicit act, and a source reconnected after being deleted // must not be dropped as a replay of the first one's schedule. func (s *Service) enqueueFirstPullSync(ctx context.Context, source *DatasourceSource) error { - return s.enqueuePullSync(ctx, source, datasourceSyncRequestSource, NewIDString()) + _, err := s.enqueuePullSync(ctx, source, datasourceSyncRequestSource, NewIDString()) + return err } // enqueueScheduledPullSync is the periodic reconcile for a provider the host @@ -1043,15 +1058,16 @@ func (s *Service) enqueueFirstPullSync(ctx context.Context, source *DatasourceSo // // STOPGAP (issue #1112). func (s *Service) enqueueScheduledPullSync(ctx context.Context, source *DatasourceSource) error { - return s.enqueuePullSync(ctx, source, DatasourceScheduledSyncSource, scheduledReconcileKey(source)) + _, err := s.enqueuePullSync(ctx, source, DatasourceScheduledSyncSource, scheduledReconcileKey(source)) + return err } // enqueuePullSync enqueues one pull-provider sync request, the job the leased // sync worker performs through RunDatasourceSync. It is exactly the job // SyncDatasourceSource enqueues for these providers; only provenance and -// idempotency key vary by caller. -func (s *Service) enqueuePullSync(ctx context.Context, source *DatasourceSource, provenance, idempotencyKey string) error { - _, err := s.datasourceJobs.EnqueueJob(ctx, &jobsv1.EnqueueJobRequest{ +// idempotency key vary by caller. It returns the job's id. +func (s *Service) enqueuePullSync(ctx context.Context, source *DatasourceSource, provenance, idempotencyKey string) (string, error) { + response, err := s.datasourceJobs.EnqueueJob(ctx, &jobsv1.EnqueueJobRequest{ Job: &jobsv1.NewJob{ Direction: jobsv1.JobDirection_JOB_DIRECTION_INBOX, Scope: &jobsv1.JobScope{Value: &jobsv1.JobScope_OrganizationId{OrganizationId: source.OrgID}}, @@ -1067,7 +1083,10 @@ func (s *Service) enqueuePullSync(ctx context.Context, source *DatasourceSource, Attributes: map[string]string{attrSourceID: source.ID}, }, }) - return err + if err != nil { + return "", err + } + return response.GetJobId(), nil } // pathInScope reports whether p is under one of the source's path prefixes at a diff --git a/module/services/accounts/code/pkg/business/datasource_file_filter.go b/module/services/accounts/code/pkg/business/datasource_file_filter.go index bc1777ae3..48ec0fa67 100644 --- a/module/services/accounts/code/pkg/business/datasource_file_filter.go +++ b/module/services/accounts/code/pkg/business/datasource_file_filter.go @@ -1,9 +1,11 @@ package business import ( + "fmt" "regexp" "strings" + "google.golang.org/genproto/googleapis/rpc/errdetails" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" ) @@ -53,3 +55,91 @@ func fileTypeAllowed(path string, extensions []string) bool { } return false } + +// Bounds on an object-storage source's key patterns. Go's regexp is RE2, so a +// tenant's expression runs in time linear in the key whatever it says; the +// bounds keep the per-key cost and the stored config small. +const ( + maxKeyPatterns = 16 + maxKeyPatternLength = 256 +) + +// keyFilterRefused is the refusal of one key-filter field: INVALID_ARGUMENT +// with a BadRequest violation naming the field, and a message that starts with +// the same path, which is what a client without the detail types reads. +func keyFilterRefused(field, reason string) error { + base := status.New(codes.InvalidArgument, field+": "+reason) + detailed, err := base.WithDetails(&errdetails.BadRequest{FieldViolations: []*errdetails.BadRequest_FieldViolation{{Field: field, Description: reason}}}) + if err != nil { + return base.Err() + } + return detailed.Err() +} + +// normalizeKeyPatterns trims, de-duplicates and compiles one pattern list, +// refusing an invalid expression with its field path and the pattern named. +func normalizeKeyPatterns(field string, values []string) ([]string, error) { + if len(values) > maxKeyPatterns { + return nil, keyFilterRefused(field, fmt.Sprintf("at most %d patterns", maxKeyPatterns)) + } + var out []string + seen := map[string]bool{} + for i, raw := range values { + value := strings.TrimSpace(raw) + path := fmt.Sprintf("%s[%d]", field, i) + switch { + case value == "": + return nil, keyFilterRefused(path, "a pattern cannot be empty") + case len(value) > maxKeyPatternLength: + return nil, keyFilterRefused(path, fmt.Sprintf("a pattern is at most %d characters", maxKeyPatternLength)) + } + if _, err := regexp.Compile(value); err != nil { + return nil, keyFilterRefused(path, fmt.Sprintf("%q is not a valid RE2 expression: %v", value, err)) + } + if !seen[value] { + out = append(out, value) + seen[value] = true + } + } + return out, nil +} + +// uploadKeyScope is an object-storage source's key filter, the InScope of its +// connector Scope as datasourceFileScope is a GitHub source's: a key is in scope +// when, with the prefix removed, it matches an include pattern (or there are +// none), matches no exclude pattern, and carries an admitted file type. The +// patterns were compiled when the config was accepted; one that no longer +// compiles admits nothing rather than everything. +func uploadKeyScope(cfg *UploadDatasourceConfig) func(key string) bool { + compile := func(patterns []string) ([]*regexp.Regexp, bool) { + out := make([]*regexp.Regexp, 0, len(patterns)) + for _, p := range patterns { + re, err := regexp.Compile(p) + if err != nil { + return nil, false + } + out = append(out, re) + } + return out, true + } + include, okInclude := compile(cfg.IncludePatterns) + exclude, okExclude := compile(cfg.ExcludePatterns) + if !okInclude || !okExclude { + return func(string) bool { return false } + } + matchesAny := func(res []*regexp.Regexp, s string) bool { + for _, re := range res { + if re.MatchString(s) { + return true + } + } + return false + } + return func(key string) bool { + relative := strings.TrimPrefix(key, cfg.Prefix) + if len(include) > 0 && !matchesAny(include, relative) { + return false + } + return !matchesAny(exclude, relative) && fileTypeAllowed(relative, cfg.FileExtensions) + } +} diff --git a/module/services/accounts/code/pkg/business/datasource_key_filter_test.go b/module/services/accounts/code/pkg/business/datasource_key_filter_test.go new file mode 100644 index 000000000..6c23928e7 --- /dev/null +++ b/module/services/accounts/code/pkg/business/datasource_key_filter_test.go @@ -0,0 +1,115 @@ +package business + +import ( + "strings" + "testing" + + "google.golang.org/genproto/googleapis/rpc/errdetails" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" +) + +// TestUploadKeyScope holds an object-storage source's key filter to its +// contract: patterns match the key with the prefix removed, unanchored and +// case-sensitive; an empty include admits everything; exclude wins; and file +// types intersect, case-insensitively, as on GitHub. +func TestUploadKeyScope(t *testing.T) { + cases := []struct { + name string + cfg UploadDatasourceConfig + admit []string + exclude []string + }{ + { + name: "no filter admits every key under the prefix", + cfg: UploadDatasourceConfig{Prefix: "kb/"}, + admit: []string{"kb/a.md", "kb/deep/b.pdf"}, + }, + { + name: "patterns see the key relative to the prefix", + cfg: UploadDatasourceConfig{Prefix: "kb/", IncludePatterns: []string{`^guides/`}}, + admit: []string{"kb/guides/a.md"}, + exclude: []string{"kb/notes/guides/a.md", "kb/a.md"}, + }, + { + name: "exclude wins over include", + cfg: UploadDatasourceConfig{Prefix: "kb/", IncludePatterns: []string{`\.md$`}, ExcludePatterns: []string{`^drafts/`}}, + admit: []string{"kb/guides/a.md"}, + exclude: []string{"kb/drafts/a.md", "kb/guides/a.pdf"}, + }, + { + name: "patterns are case-sensitive unless they fold case", + cfg: UploadDatasourceConfig{Prefix: "kb/", IncludePatterns: []string{`^Policies/`, `(?i)^handbook/`}}, + admit: []string{"kb/Policies/a.md", "kb/HANDBOOK/a.md"}, + exclude: []string{"kb/policies/a.md"}, + }, + { + name: "file types intersect with the patterns, case-insensitively", + cfg: UploadDatasourceConfig{Prefix: "kb/", IncludePatterns: []string{`^guides/`}, FileExtensions: []string{".md"}}, + admit: []string{"kb/guides/A.MD"}, + exclude: []string{"kb/guides/a.pdf", "kb/other/a.md"}, + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + inScope := uploadKeyScope(&tc.cfg) + for _, key := range tc.admit { + if !inScope(key) { + t.Errorf("%q is excluded, want admitted", key) + } + } + for _, key := range tc.exclude { + if inScope(key) { + t.Errorf("%q is admitted, want excluded", key) + } + } + }) + } +} + +// TestNormalizeKeyPatternsRefusesByName: every refusal is INVALID_ARGUMENT and +// names the offending element, in the message a client reads and in the +// BadRequest violation a client can decode. +func TestNormalizeKeyPatternsRefusesByName(t *testing.T) { + tooMany := make([]string, maxKeyPatterns+1) + for i := range tooMany { + tooMany[i] = "a" + } + cases := []struct { + name string + patterns []string + field string + mention string + }{ + {"an invalid RE2 expression", []string{`\.md$`, `(unclosed`}, "upload.include_patterns[1]", `"(unclosed"`}, + {"a backreference RE2 does not support", []string{`(a)\1`}, "upload.include_patterns[0]", `"(a)\\1"`}, + {"an empty pattern", []string{" "}, "upload.include_patterns[0]", "empty"}, + {"a pattern over the length cap", []string{strings.Repeat("a", maxKeyPatternLength+1)}, "upload.include_patterns[0]", "256"}, + {"more patterns than the cap", tooMany, "upload.include_patterns", "16"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + _, err := normalizeKeyPatterns("upload.include_patterns", tc.patterns) + st := status.Convert(err) + if err == nil || st.Code() != codes.InvalidArgument { + t.Fatalf("err = %v, want INVALID_ARGUMENT", err) + } + if !strings.HasPrefix(st.Message(), tc.field+": ") || !strings.Contains(st.Message(), tc.mention) { + t.Fatalf("message %q does not name %s and %s", st.Message(), tc.field, tc.mention) + } + var field string + for _, d := range st.Details() { + if br, ok := d.(*errdetails.BadRequest); ok && len(br.GetFieldViolations()) == 1 { + field = br.GetFieldViolations()[0].GetField() + } + } + if field != tc.field { + t.Fatalf("BadRequest field = %q, want %q", field, tc.field) + } + }) + } + got, err := normalizeKeyPatterns("upload.exclude_patterns", []string{" ^drafts/ ", "^drafts/", `\.tmp$`}) + if err != nil || strings.Join(got, ",") != `^drafts/,\.tmp$` { + t.Fatalf("normalize = %v, %v; want trimmed and de-duplicated", got, err) + } +} diff --git a/module/services/accounts/code/pkg/business/datasource_object_storage_admission_test.go b/module/services/accounts/code/pkg/business/datasource_object_storage_admission_test.go new file mode 100644 index 000000000..1556ee877 --- /dev/null +++ b/module/services/accounts/code/pkg/business/datasource_object_storage_admission_test.go @@ -0,0 +1,102 @@ +package business_test + +import ( + "context" + "net" + "strings" + "testing" + + storagev0 "github.com/codefly-dev/service-object-storage/gen/codefly/storage/v0" + "google.golang.org/grpc" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/credentials/insecure" + "google.golang.org/grpc/status" + "google.golang.org/grpc/test/bufconn" + + "accounts/pkg/business" +) + +// composedStorage answers Capabilities as a composed object-storage service +// does: the released 0.0.18 knows no binding and no batched read, so it +// reports neither; a release with #108's binding reports its batch size. +type composedStorage struct { + storagev0.UnimplementedObjectStorageServer + binding bool +} + +func (c composedStorage) Capabilities(context.Context, *storagev0.CapabilitiesRequest) (*storagev0.BackendCapabilities, error) { + if !c.binding { + return &storagev0.BackendCapabilities{Backend: "minio"}, nil + } + return &storagev0.BackendCapabilities{Backend: "minio", MaxObjectBytes: 5 << 20, MaxListLimit: 1000, MaxGetManyItems: 1000}, nil +} + +func composedStorageClient(t *testing.T, binding bool) storagev0.ObjectStorageClient { + t.Helper() + lis := bufconn.Listen(1 << 20) + srv := grpc.NewServer() + storagev0.RegisterObjectStorageServer(srv, composedStorage{binding: binding}) + go func() { _ = srv.Serve(lis) }() + t.Cleanup(srv.Stop) + cc, err := grpc.NewClient("passthrough:///composed", + grpc.WithContextDialer(func(ctx context.Context, _ string) (net.Conn, error) { return lis.DialContext(ctx) }), + grpc.WithTransportCredentials(insecure.NewCredentials())) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = cc.Close() }) + return storagev0.NewObjectStorageClient(cc) +} + +// TestObjectStorageIsRefusedAtAdmissionWhenTheComposedServiceCannotServeIt is +// review r1's major 3, pinned where the defect was: with the released 0.0.18 +// composed — a service that answers, but cannot serve a binding — the catalog +// accepted new object-storage sources and AddSource stored them, and only each +// sync then failed. Wiring that service is refused, and the refusal is at +// admission: the catalog does not offer the provider, AddSource refuses by +// name, and nothing is stored or audited. A service that serves the binding is +// admitted. +func TestObjectStorageIsRefusedAtAdmissionWhenTheComposedServiceCannotServeIt(t *testing.T) { + ctx := context.Background() + accepts := func(svc *business.Service) bool { + for _, e := range svc.DatasourceCatalog() { + if e.Descriptor.Key == business.DatasourceProviderUpload { + return e.AcceptsNewSources + } + } + t.Fatal("object storage is not in the catalog") + return false + } + input := business.AddSourceInput{ + OrgID: testOrg, Provider: business.DatasourceProviderUpload, CollectionLabel: "guides", Credential: "secretkey", Upload: uploadConfig(), + } + + store := newDatasourceFakeStore() + svc, audit := newDatasourceService(store, &recordingProducer{}, nil) + if err := svc.SetObjectStorage(ctx, composedStorageClient(t, false)); err == nil { + t.Fatal("wiring a composed service that cannot serve a binding succeeded") + } + if accepts(svc) { + t.Fatal("the catalog accepts object-storage sources with a composed service that cannot serve them") + } + _, err := svc.AddSource(ctx, "actor-1", input) + if status.Code(err) != codes.FailedPrecondition || !strings.Contains(err.Error(), business.ErrObjectStorageUnavailable.Error()) { + t.Fatalf("AddSource with a service that cannot serve a binding = %v, want the unavailable refusal", err) + } + if sources, err := svc.ListDatasourceSources(ctx, testOrg); err != nil || len(sources) != 0 { + t.Fatalf("a refused connect stored %v (%v)", sources, err) + } + if got := audit.types(); len(got) != 0 { + t.Fatalf("a refused connect recorded %v", got) + } + + if err := svc.SetObjectStorage(ctx, composedStorageClient(t, true)); err != nil { + t.Fatalf("wiring a service that serves the binding = %v", err) + } + if !accepts(svc) { + t.Fatal("the catalog refuses object-storage sources with a service that serves them") + } + if _, err := svc.AddSource(ctx, "actor-1", input); err != nil { + t.Fatalf("AddSource with a service that serves the binding = %v", err) + } +} diff --git a/module/services/accounts/code/pkg/business/datasource_update.go b/module/services/accounts/code/pkg/business/datasource_update.go index 05675e6c2..4b81c28cb 100644 --- a/module/services/accounts/code/pkg/business/datasource_update.go +++ b/module/services/accounts/code/pkg/business/datasource_update.go @@ -25,6 +25,9 @@ type DatasourceFetchScope struct { Branch string Paths []string FileExtensions []string + // An object-storage source's key patterns; FileExtensions is its file types. + IncludePatterns []string + ExcludePatterns []string } // UpdateSourceInput is what a connected source may be re-aimed at: its branch @@ -36,6 +39,10 @@ type UpdateSourceInput struct { Branch string Paths []string FileExtensions []string + // IncludePatterns and ExcludePatterns are an object-storage source's key + // patterns, replaced whole like the rest. + IncludePatterns []string + ExcludePatterns []string // Expected is the scope the caller was shown, which the edit is accepted // against. Required, and nil is refused rather than read as "apply it // anyway": a whole-field replacement applied unconditionally is precisely @@ -150,8 +157,14 @@ func (s *Service) UpdateSource(ctx context.Context, actorID string, input Update // providers are configured by their own fields (a base URL, a sitemap, a // bucket prefix), and silently accepting a branch for one of them would // report a change that changed nothing. + if source.Provider == DatasourceProviderUpload { + return s.updateUploadKeyFilter(ctx, actorID, source, input) + } if source.Provider != DatasourceProviderGitHub { - return nil, "", status.Error(codes.FailedPrecondition, "only a github source has a branch and path filters") + return nil, "", status.Error(codes.FailedPrecondition, "only a github or object-storage source has filters to edit") + } + if len(input.IncludePatterns) > 0 || len(input.ExcludePatterns) > 0 { + return nil, "", keyFilterRefused("include_patterns", "a github source has no key patterns") } branch := strings.TrimSpace(input.Branch) @@ -251,6 +264,95 @@ func (s *Service) UpdateSource(ctx context.Context, actorID string, input Update return &updated, jobID, nil } +// sameUploadKeyFilter reports whether scope is the key filter cfg holds, after +// the same normalization comparison sameFetchScope applies. +func sameUploadKeyFilter(scope DatasourceFetchScope, cfg *UploadDatasourceConfig) bool { + trimmed := func(values []string) []string { + var out []string + for _, v := range values { + if v = strings.TrimSpace(v); v != "" && !slices.Contains(out, v) { + out = append(out, v) + } + } + return out + } + return slices.Equal(trimmed(scope.IncludePatterns), trimmed(cfg.IncludePatterns)) && + slices.Equal(trimmed(scope.ExcludePatterns), trimmed(cfg.ExcludePatterns)) && + slices.Equal(comparableFileExtensions(scope.FileExtensions), comparableFileExtensions(cfg.FileExtensions)) +} + +// updateUploadKeyFilter is UpdateSource for an object-storage source: it +// replaces the key patterns and file types and nothing else (the bucket and +// prefix are the source's identity), under the same lock, expectation, audit +// and reconfigured sync as a github edit. The next sync lists through the new +// filter, so a narrowed filter's closing listing omits — and the store +// tombstones — every key it now excludes. +func (s *Service) updateUploadKeyFilter(ctx context.Context, actorID string, source *DatasourceSource, input UpdateSourceInput) (*DatasourceSource, string, error) { + w := wool.Get(ctx).In("updateUploadKeyFilter") + if strings.TrimSpace(input.Branch) != "" || len(input.Paths) > 0 { + return nil, "", keyFilterRefused("paths", "an object-storage source has no branch or paths; it is scoped by its prefix and key patterns") + } + var next UploadDatasourceConfig + if err := normalizeUploadKeyFilter(&next, &UploadDatasourceConfig{ + IncludePatterns: input.IncludePatterns, ExcludePatterns: input.ExcludePatterns, FileExtensions: input.FileExtensions, + }, ""); err != nil { + return nil, "", err + } + nextScope := DatasourceFetchScope{IncludePatterns: next.IncludePatterns, ExcludePatterns: next.ExcludePatterns, FileExtensions: next.FileExtensions} + var ( + updated DatasourceSource + changed bool + ) + if err := s.store.WithOrgTx(ctx, source.OrgID, func(ctx context.Context) error { + current, err := s.store.LockDatasourceSourceFetchScope(ctx, source.OrgID, source.ID) + if err != nil { + return err + } + if current == nil || current.Upload == nil { + return ErrDatasourceSourceNotFound + } + if !sameUploadKeyFilter(*input.Expected, current.Upload) { + return ErrDatasourceSourceScopeChanged + } + updated = *current + updated.BoundaryLabel = source.BoundaryLabel + if sameUploadKeyFilter(nextScope, current.Upload) { + return nil + } + cfg := *current.Upload + cfg.IncludePatterns, cfg.ExcludePatterns, cfg.FileExtensions = next.IncludePatterns, next.ExcludePatterns, next.FileExtensions + updated.Upload = &cfg + if err := s.store.UpdateDatasourceSourceFetchScope(ctx, &updated); err != nil { + return err + } + changed = true + return s.emitTx(ctx, actorID, "user", EventDatasourceSourceUpdated, "datasource", updated.ID, updated.OrgID, map[string]any{ + "bucket": cfg.Bucket, + "prefix": cfg.Prefix, + "include_patterns": cfg.IncludePatterns, + "exclude_patterns": cfg.ExcludePatterns, + "file_extensions": cfg.FileExtensions, + "boundary": updated.BoundaryNodeID, + }) + }); err != nil { + if errors.Is(err, ErrDatasourceSourceNotFound) || errors.Is(err, ErrDatasourceSourceScopeChanged) { + return nil, "", err + } + return nil, "", w.Wrapf(err, "persist datasource source update") + } + if !changed { + return &updated, "", nil + } + // A pull source re-syncs through its own engine, not GitHub's reconcile. + jobID, err := s.enqueuePullSync(ctx, &updated, datasourceSyncRequestSource, NewIDString()) + if err != nil { + w.Warn("enqueue re-sync failed; the periodic reconcile will re-sync the source", + wool.Field("source", updated.ID), wool.ErrField(err)) + return &updated, "", nil + } + return &updated, jobID, nil +} + // enqueueReconfiguredSync queues the re-snapshot an update owes the consuming // module: the same forced reconcile "Sync now" and the first sync at connect // enqueue, in the mode that names why it ran. diff --git a/module/services/accounts/code/pkg/business/datasource_upload_budget_test.go b/module/services/accounts/code/pkg/business/datasource_upload_budget_test.go new file mode 100644 index 000000000..5bf91f947 --- /dev/null +++ b/module/services/accounts/code/pkg/business/datasource_upload_budget_test.go @@ -0,0 +1,161 @@ +package business_test + +import ( + "context" + "strings" + "sync" + "testing" + "time" + + "accounts/pkg/business" + jobsv1 "accounts/pkg/gen/saas/jobs/v1" +) + +// recordingBudgets is a meter that always admits and records what was spent: +// under which credential key, and against which limit — the whole quota for an +// interactive operation, the background share otherwise. +type recordingBudgets struct { + mu sync.Mutex + spends []budgetSpend +} + +type budgetSpend struct { + key string + limit int +} + +func (r *recordingBudgets) SpendDatasourceBudget(_ context.Context, key string, limit int, window time.Duration, now time.Time) (bool, time.Time, time.Time, error) { + r.mu.Lock() + defer r.mu.Unlock() + r.spends = append(r.spends, budgetSpend{key, limit}) + return true, now.Add(window), time.Time{}, nil +} + +func (r *recordingBudgets) BlockDatasourceBudget(context.Context, string, time.Time) error { + return nil +} + +func (r *recordingBudgets) take() []budgetSpend { + r.mu.Lock() + defer r.mu.Unlock() + out := r.spends + r.spends = nil + return out +} + +// TestUploadSourcesOnOneCredentialSpendOneBudget holds object storage to the +// scheduler's contract: one provider credential's quota is shared by all its +// sources. Two sources reading with the same access key at the same store — +// two audiences of one bucket, which the descriptor recommends — spend one +// budget; a third with another key spends its own. And a sync a person asked +// for may spend the whole quota, while the scheduled pass spends only the +// background share. +func TestUploadSourcesOnOneCredentialSpendOneBudget(t *testing.T) { + producer := &recordingProducer{} + svc, _ := newDatasourceService(newDatasourceFakeStore(), producer, nil) + // No delivered link key: the grouping must not depend on host configuration. + budgets := &recordingBudgets{} + svc.SetDatasourceBudgetStore(budgets) + useUploadBucket(svc, &fakeUploadBucket{objects: []fakeObject{{key: "kb/a.md", etag: "a", body: "A"}}}) + add := func(prefix, keyID string) *business.DatasourceSource { + cfg := uploadConfig() + cfg.Prefix, cfg.AccessKeyID = prefix, keyID + source, err := svc.AddSource(context.Background(), "actor-1", business.AddSourceInput{ + OrgID: testOrg, Provider: business.DatasourceProviderUpload, CollectionLabel: "guides-" + prefix + keyID, Credential: "secretkey", Upload: cfg, + }) + if err != nil { + t.Fatal(err) + } + return source + } + keyOf := func(source *business.DatasourceSource) string { + budgets.take() + if _, err := svc.RunDatasourceSync(context.Background(), source.ID); err != nil { + t.Fatal(err) + } + spends := budgets.take() + if len(spends) == 0 { + t.Fatalf("a sync of %s spent no budget", source.ID) + } + for _, s := range spends[1:] { + if s.key != spends[0].key { + t.Fatalf("one sync spent under two keys: %v", spends) + } + } + return spends[0].key + } + staff, interns, other := add("kb/", "AKIASHARED"), add("kb/interns/", "AKIASHARED"), add("kb/", "AKIAOTHER") + if a, b := keyOf(staff), keyOf(interns); a != b || a == "" { + t.Fatalf("two sources on one credential spent under %q and %q, want one key", a, b) + } + if keyOf(staff) == keyOf(other) { + t.Fatal("sources on different credentials share a budget") + } + if key := keyOf(staff); !strings.HasPrefix(key, "upload:credential:") || strings.Contains(key, "AKIASHARED") { + t.Fatalf("budget key %q, want a digest that names no credential in the clear", key) + } + + handler := svc.NewDatasourceSyncJobHandler() + if _, err := svc.SyncDatasourceSource(context.Background(), "actor-1", testOrg, staff.ID); err != nil { + t.Fatal(err) + } + requested := producer.jobs[len(producer.jobs)-1] + limitOf := func(source string) int { + budgets.take() + envelope := &jobsv1.JobEnvelope{Queue: requested.GetQueue(), Topic: requested.GetTopic(), Source: source, Attributes: requested.GetAttributes()} + if err := handler(context.Background(), envelope); err != nil { + t.Fatal(err) + } + spends := budgets.take() + if len(spends) == 0 { + t.Fatal("the sync spent no budget") + } + return spends[0].limit + } + interactive, background := limitOf(requested.GetSource()), limitOf(business.DatasourceScheduledSyncSource) + if interactive <= background { + t.Fatalf("a requested sync spent against %d and the scheduled pass against %d, want the requested one to reach the reserved share", interactive, background) + } +} + +// TestUploadSourcesConnectedBeforeGroupingShareOneBudget: review r2's g1. A +// source stored before credential grouping existed — or connected on a host +// with no delivered link key — carries nothing but its endpoint and access key +// id. Its budget key is derived from those on every use, so two such sources on +// one credential spend one quota and honour one provider block, with no +// backfill. A source whose store spells its host differently in case is the +// same credential. +func TestUploadSourcesConnectedBeforeGroupingShareOneBudget(t *testing.T) { + store := newDatasourceFakeStore() + svc, _ := newDatasourceService(store, &recordingProducer{}, nil) + budgets := &recordingBudgets{} + svc.SetDatasourceBudgetStore(budgets) + useUploadBucket(svc, &fakeUploadBucket{objects: []fakeObject{{key: "kb/a.md", etag: "a", body: "A"}}}) + add := func(label, endpoint string) *business.DatasourceSource { + cfg := uploadConfig() + cfg.Endpoint = endpoint + source, err := svc.AddExistingSource(context.Background(), "actor-1", business.AddSourceInput{ + OrgID: testOrg, Provider: business.DatasourceProviderUpload, CollectionLabel: label, Credential: "secretkey", Upload: cfg, + }) + if err != nil { + t.Fatal(err) + } + return source + } + a, b := add("legacy a", "https://s3.us-east-1.amazonaws.com"), add("legacy b", "https://S3.US-EAST-1.AMAZONAWS.COM") + var keys []string + for _, src := range []*business.DatasourceSource{a, b} { + budgets.take() + if _, err := svc.RunDatasourceSync(context.Background(), src.ID); err != nil { + t.Fatal(err) + } + spends := budgets.take() + if len(spends) == 0 { + t.Fatal("a sync spent no budget") + } + keys = append(keys, spends[0].key) + } + if keys[0] != keys[1] || strings.HasPrefix(keys[0], "upload:source:") { + t.Fatalf("two sources on one credential spent under %q and %q, want one credential key", keys[0], keys[1]) + } +} diff --git a/module/services/accounts/code/pkg/business/datasource_upload_key_filter_test.go b/module/services/accounts/code/pkg/business/datasource_upload_key_filter_test.go new file mode 100644 index 000000000..1ee834438 --- /dev/null +++ b/module/services/accounts/code/pkg/business/datasource_upload_key_filter_test.go @@ -0,0 +1,201 @@ +package business_test + +import ( + "context" + "reflect" + "sort" + "strings" + "testing" + + "accounts/pkg/business" + + "google.golang.org/genproto/googleapis/rpc/errdetails" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" +) + +// refusedField is the field an INVALID_ARGUMENT refusal names, read from its +// BadRequest violation, after checking the message leads with the same path. +func refusedField(t *testing.T, err error) string { + t.Helper() + st := status.Convert(err) + if err == nil || st.Code() != codes.InvalidArgument { + t.Fatalf("err = %v, want INVALID_ARGUMENT", err) + } + for _, d := range st.Details() { + if br, ok := d.(*errdetails.BadRequest); ok && len(br.GetFieldViolations()) == 1 { + field := br.GetFieldViolations()[0].GetField() + if !strings.HasPrefix(st.Message(), field+": ") { + t.Fatalf("message %q does not lead with the field %q", st.Message(), field) + } + return field + } + } + t.Fatalf("refusal %v carries no BadRequest violation", err) + return "" +} + +// An invalid key filter is refused when the source is added — by name, and +// before anything is stored — never discovered at sync. +func TestAddSource_UploadRefusesAnInvalidKeyFilterByField(t *testing.T) { + cases := map[string]struct { + mutate func(*business.UploadDatasourceConfig) + field string + }{ + "invalid include": {func(c *business.UploadDatasourceConfig) { c.IncludePatterns = []string{`ok`, `[z-a]`} }, "upload.include_patterns[1]"}, + "invalid exclude": {func(c *business.UploadDatasourceConfig) { c.ExcludePatterns = []string{`^(?!drafts/)`} }, "upload.exclude_patterns[0]"}, + "invalid file type": {func(c *business.UploadDatasourceConfig) { c.FileExtensions = []string{"*.md"} }, "upload.file_extensions"}, + } + for name, tc := range cases { + t.Run(name, func(t *testing.T) { + store := newDatasourceFakeStore() + svc, _ := newDatasourceService(store, &recordingProducer{}, nil) + cfg := uploadConfig() + tc.mutate(cfg) + _, err := svc.AddExistingSource(context.Background(), "actor-1", business.AddSourceInput{ + OrgID: testOrg, Provider: business.DatasourceProviderUpload, CollectionLabel: "guides", + Credential: "secretkey", Upload: cfg, + }) + if got := refusedField(t, err); got != tc.field { + t.Fatalf("refused field = %q, want %q", got, tc.field) + } + if sources, _ := svc.ListDatasourceSources(context.Background(), testOrg); len(sources) != 0 { + t.Fatalf("a refused source was stored: %+v", sources) + } + }) + } +} + +// filteredUploadSource connects an object-storage source with a key filter. +func filteredUploadSource(t *testing.T, svc *business.Service, include, exclude []string) *business.DatasourceSource { + t.Helper() + cfg := uploadConfig() + cfg.IncludePatterns, cfg.ExcludePatterns = include, exclude + // The sync test lists four keys, three of them matches: an excluded key + // that used up the cap would leave its listing incomplete. + cfg.MaxObjects = 3 + source, err := svc.AddExistingSource(context.Background(), "actor-1", business.AddSourceInput{ + OrgID: testOrg, Provider: business.DatasourceProviderUpload, CollectionLabel: "guides", + Credential: "secretkey", Upload: cfg, + }) + if err != nil { + t.Fatal(err) + } + return source +} + +func uploadScopeOf(source *business.DatasourceSource) *business.DatasourceFetchScope { + return &business.DatasourceFetchScope{ + IncludePatterns: source.Upload.IncludePatterns, + ExcludePatterns: source.Upload.ExcludePatterns, + FileExtensions: source.Upload.FileExtensions, + } +} + +func TestUpdateSource_UploadRefusesAnInvalidKeyFilterByField(t *testing.T) { + store := newDatasourceFakeStore() + svc, audit := newDatasourceService(store, &recordingProducer{}, nil) + source := filteredUploadSource(t, svc, []string{`^guides/`}, nil) + + _, _, err := svc.UpdateSource(context.Background(), "actor-1", business.UpdateSourceInput{ + OrgID: testOrg, ID: source.ID, IncludePatterns: []string{`^guides/`}, ExcludePatterns: []string{`a{1001}`}, + Expected: uploadScopeOf(source), + }) + if got := refusedField(t, err); got != "exclude_patterns[0]" { + t.Fatalf("refused field = %q, want exclude_patterns[0]", got) + } + _, _, err = svc.UpdateSource(context.Background(), "actor-1", business.UpdateSourceInput{ + OrgID: testOrg, ID: source.ID, Paths: []string{"docs/"}, Expected: uploadScopeOf(source), + }) + if got := refusedField(t, err); got != "paths" { + t.Fatalf("refused field = %q, want paths: a bucket source has no paths", got) + } + stored, err := svc.GetDatasourceSource(context.Background(), testOrg, source.ID) + if err != nil || !reflect.DeepEqual(stored.Upload.ExcludePatterns, []string(nil)) { + t.Fatalf("a refused edit was written: %+v, %v", stored.Upload, err) + } + if entries := audit.entriesOf(business.EventDatasourceSourceUpdated); len(entries) != 0 { + t.Fatalf("updated events = %d, want 0 for a refused edit", len(entries)) + } +} + +func listedKeys(t *testing.T, l mirrorListing) []string { + t.Helper() + var keys []string + for _, f := range *l.Files { + keys = append(keys, f.Key) + } + sort.Strings(keys) + return keys +} + +func objectKeys(p *recordingProducer) []string { + var keys []string + for _, job := range uploadObjectJobs(p) { + keys = append(keys, job.GetAttributes()["upload.key"]) + } + sort.Strings(keys) + return keys +} + +// The mixed listing on the mirror seam: only matching keys are enqueued, an +// excluded key is never fetched, and the closing listing names only matches. +// Narrowing the filter through UpdateSource then re-syncs, and that sync's +// closing listing drops exactly the keys the narrower filter excludes — which +// is what tombstones them in the store — and nothing else. +func TestRunDatasourceSync_UploadKeyFilterNarrowingTombstonesOnlyTheExcluded(t *testing.T) { + producer := &recordingProducer{} + svc, audit := newDatasourceService(newDatasourceFakeStore(), producer, nil) + fake := &fakeUploadBucket{objects: []fakeObject{ + {key: "kb/guides/a.md", etag: "a", body: "a"}, {key: "kb/guides/b.pdf", etag: "b", body: "b"}, + {key: "kb/drafts/c.md", etag: "c", body: "c"}, {key: "kb/notes/d.md", etag: "d", body: "d"}, + }} + useUploadBucket(svc, fake) + source := filteredUploadSource(t, svc, []string{`\.md$`, `^guides/`}, []string{`^drafts/`}) + takeConnectSync(t, producer, source) + + if _, err := svc.RunDatasourceSync(context.Background(), source.ID); err != nil { + t.Fatal(err) + } + wide := []string{"kb/guides/a.md", "kb/guides/b.pdf", "kb/notes/d.md"} + if got := objectKeys(producer); !reflect.DeepEqual(got, wide) { + t.Fatalf("enqueued = %v, want %v", got, wide) + } + sort.Strings(fake.read) + if !reflect.DeepEqual(fake.read, wide) { + t.Fatalf("read = %v, want only the matches %v: an excluded key is never read", fake.read, wide) + } + if listings := uploadListings(t, producer); len(listings) != 1 || !reflect.DeepEqual(listedKeys(t, listings[0]), wide) { + t.Fatalf("closing listing does not name exactly the matches %v", wide) + } + + // Narrow: exclude the notes folder as well. + updated, jobID, err := svc.UpdateSource(context.Background(), "actor-1", business.UpdateSourceInput{ + OrgID: testOrg, ID: source.ID, + IncludePatterns: []string{`\.md$`, `^guides/`}, ExcludePatterns: []string{`^drafts/`, `^notes/`}, + Expected: uploadScopeOf(source), + }) + if err != nil || jobID == "" { + t.Fatalf("narrowing = %v, job %q; want a re-sync", err, jobID) + } + if entries := audit.entriesOf(business.EventDatasourceSourceUpdated); len(entries) != 1 { + t.Fatalf("updated events = %d, want 1", len(entries)) + } + if _, _, err := svc.UpdateSource(context.Background(), "actor-1", business.UpdateSourceInput{ + OrgID: testOrg, ID: source.ID, IncludePatterns: []string{`.*`}, Expected: uploadScopeOf(source), + }); err != business.ErrDatasourceSourceScopeChanged { + t.Fatalf("an edit against the scope before the narrowing = %v, want ErrDatasourceSourceScopeChanged", err) + } + + producer.mu.Lock() + producer.jobs = nil + producer.mu.Unlock() + if _, err := svc.RunDatasourceSync(context.Background(), updated.ID); err != nil { + t.Fatal(err) + } + narrow := []string{"kb/guides/a.md", "kb/guides/b.pdf"} + listings := uploadListings(t, producer) + if len(listings) != 1 || !reflect.DeepEqual(listedKeys(t, listings[0]), narrow) { + t.Fatalf("closing listing after narrowing does not name exactly %v: kb/notes/d.md must be the only key dropped", narrow) + } +} diff --git a/module/services/accounts/code/pkg/business/datasource_upload_mirror_test.go b/module/services/accounts/code/pkg/business/datasource_upload_mirror_test.go index bfdba86ef..761ec4c80 100644 --- a/module/services/accounts/code/pkg/business/datasource_upload_mirror_test.go +++ b/module/services/accounts/code/pkg/business/datasource_upload_mirror_test.go @@ -56,14 +56,10 @@ func uploadObjectJobs(p *recordingProducer) []*jobsv1.NewJob { return out } -func mirrorClient() *fakeUploadClient { - return &fakeUploadClient{ - entries: []objectstore.Entry{{Key: "loans/a.xlsx", ETag: "list-a"}, {Key: "loans/", ETag: "dir"}, {Key: "loans/b.xlsx", ETag: "list-b"}}, - objects: map[string]objectstore.Object{ - "loans/a.xlsx": {Key: "loans/a.xlsx", Body: []byte("A"), ETag: "get-a"}, - "loans/b.xlsx": {Key: "loans/b.xlsx", Body: []byte("B"), ETag: "get-b"}, - }, - } +func mirrorBucket() *fakeUploadBucket { + return &fakeUploadBucket{objects: []fakeObject{ + {key: "loans/a.xlsx", etag: "get-a", body: "A"}, {key: "loans/", etag: "dir"}, {key: "loans/b.xlsx", etag: "get-b", body: "B"}, + }} } // TestRunDatasourceSync_UploadClosesWithTheCompleteListing pins the mirror: a @@ -73,8 +69,7 @@ func mirrorClient() *fakeUploadClient { func TestRunDatasourceSync_UploadClosesWithTheCompleteListing(t *testing.T) { producer := &recordingProducer{} svc, _ := newDatasourceService(newDatasourceFakeStore(), producer, nil) - fake := mirrorClient() - svc.SetDatasourceUploadClientFactory(func(business.UploadDatasourceConfig, string) business.UploadContentClient { return fake }) + useUploadBucket(svc, mirrorBucket()) source := newUploadSource(t, svc) if _, err := svc.RunDatasourceSync(context.Background(), source.ID); err != nil { @@ -128,8 +123,7 @@ func TestRunDatasourceSync_UploadClosesWithTheCompleteListing(t *testing.T) { func TestRunDatasourceSync_UploadNextSyncRedeliversUnderANewOrdinal(t *testing.T) { producer := &recordingProducer{} svc, _ := newDatasourceService(newDatasourceFakeStore(), producer, nil) - fake := mirrorClient() - svc.SetDatasourceUploadClientFactory(func(business.UploadDatasourceConfig, string) business.UploadContentClient { return fake }) + useUploadBucket(svc, mirrorBucket()) source := newUploadSource(t, svc) for i := 0; i < 2; i++ { if _, err := svc.RunDatasourceSync(context.Background(), source.ID); err != nil { @@ -155,8 +149,7 @@ func TestRunDatasourceSync_UploadNextSyncRedeliversUnderANewOrdinal(t *testing.T func TestRunDatasourceSync_UploadEmptiedFolderListsNothingExplicitly(t *testing.T) { producer := &recordingProducer{} svc, _ := newDatasourceService(newDatasourceFakeStore(), producer, nil) - fake := &fakeUploadClient{} - svc.SetDatasourceUploadClientFactory(func(business.UploadDatasourceConfig, string) business.UploadContentClient { return fake }) + useUploadBucket(svc, &fakeUploadBucket{}) source := newUploadSource(t, svc) if _, err := svc.RunDatasourceSync(context.Background(), source.ID); err != nil { t.Fatal(err) @@ -168,15 +161,17 @@ func TestRunDatasourceSync_UploadEmptiedFolderListsNothingExplicitly(t *testing. } // TestRunDatasourceSync_UploadFailedOrTruncatedListingNeverMirrors pins fail -// closed: a listing error sends nothing, and a truncated listing still -// delivers what it listed but sends no listing and fails the sync terminally. +// closed: a listing error sends nothing; a listing past the source's object cap +// delivers nothing, sends no listing and fails the sync terminally; and an +// object changed between the listing and its read fails the sync without a +// listing, for the next one to list afresh. func TestRunDatasourceSync_UploadFailedOrTruncatedListingNeverMirrors(t *testing.T) { t.Run("listing error", func(t *testing.T) { producer := &recordingProducer{} svc, _ := newDatasourceService(newDatasourceFakeStore(), producer, nil) - fake := mirrorClient() + fake := mirrorBucket() fake.listErr = errors.New("AccessDenied") - svc.SetDatasourceUploadClientFactory(func(business.UploadDatasourceConfig, string) business.UploadContentClient { return fake }) + useUploadBucket(svc, fake) source := newUploadSource(t, svc) // Connecting enqueues the source's first sync request; this subtest counts // everything on the producer, so that one is taken out of the way first. @@ -188,23 +183,26 @@ func TestRunDatasourceSync_UploadFailedOrTruncatedListingNeverMirrors(t *testing t.Fatalf("a failed listing enqueued %d job(s)", len(producer.jobs)) } }) - t.Run("truncated listing", func(t *testing.T) { + t.Run("listing past the object cap", func(t *testing.T) { producer := &recordingProducer{} - store := newDatasourceFakeStore() - svc, _ := newDatasourceService(store, producer, nil) - fake := mirrorClient() - fake.incomplete = true - svc.SetDatasourceUploadClientFactory(func(business.UploadDatasourceConfig, string) business.UploadContentClient { return fake }) - source := newUploadSource(t, svc) - _, err := svc.RunDatasourceSync(context.Background(), source.ID) - if !errors.Is(err, business.ErrUploadListingIncomplete) { - t.Fatalf("truncated listing: err = %v, want ErrUploadListingIncomplete", err) + svc, _ := newDatasourceService(newDatasourceFakeStore(), producer, nil) + fake := mirrorBucket() + useUploadBucket(svc, fake) + cfg := uploadConfig() + cfg.MaxObjects = 1 + source, err := svc.AddExistingSource(context.Background(), "actor-1", business.AddSourceInput{ + OrgID: testOrg, Provider: business.DatasourceProviderUpload, CollectionLabel: "guides", Credential: "secretkey", Upload: cfg, + }) + if err != nil { + t.Fatal(err) } - if len(uploadObjectJobs(producer)) != 2 { - t.Fatal("a truncated listing stopped delivering the objects it did list") + takeConnectSync(t, producer, source) + _, err = svc.RunDatasourceSync(context.Background(), source.ID) + if !errors.Is(err, business.ErrUploadListingIncomplete) { + t.Fatalf("listing past the cap: err = %v, want ErrUploadListingIncomplete", err) } - if listings := uploadListings(t, producer); len(listings) != 0 { - t.Fatalf("a truncated listing was sent as complete: %+v", listings) + if len(producer.jobs) != 0 || len(fake.read) != 0 { + t.Fatalf("a listing past the cap enqueued %d job(s) and read %v; it must deliver nothing", len(producer.jobs), fake.read) } if _, err := svc.SyncDatasourceSource(context.Background(), "actor-1", testOrg, source.ID); err != nil { t.Fatal(err) @@ -216,22 +214,40 @@ func TestRunDatasourceSync_UploadFailedOrTruncatedListingNeverMirrors(t *testing t.Fatalf("handler error = %v, want a terminal ProcessingError", jobErr) } if listings := uploadListings(t, producer); len(listings) != 0 { - t.Fatal("the job handler's sync sent a listing of a truncated folder") + t.Fatal("the job handler's sync sent a listing of a folder past its cap") } }) - t.Run("vanished object is left out of the listing", func(t *testing.T) { + // The store's bucket gone between the listing and the reads: the service + // answers every object of the batch NOT_FOUND, which ServiceBucket reports + // as ErrObjectChanged. Tombstone authority is the closing listing alone, and + // it is sent only after a successful listing in the same sync and the + // delivery of every object in it — so this sync fails and deletes nothing. + t.Run("every object gone after its listing", func(t *testing.T) { producer := &recordingProducer{} svc, _ := newDatasourceService(newDatasourceFakeStore(), producer, nil) - fake := mirrorClient() - fake.fetchErr = map[string]error{"loans/b.xlsx": objectstore.ErrObjectNotFound} - svc.SetDatasourceUploadClientFactory(func(business.UploadDatasourceConfig, string) business.UploadContentClient { return fake }) + fake := mirrorBucket() + fake.readErr = map[string]error{"loans/a.xlsx": objectstore.ErrObjectChanged, "loans/b.xlsx": objectstore.ErrObjectChanged} + useUploadBucket(svc, fake) source := newUploadSource(t, svc) - if _, err := svc.RunDatasourceSync(context.Background(), source.ID); err != nil { - t.Fatal(err) + if _, err := svc.RunDatasourceSync(context.Background(), source.ID); err == nil { + t.Fatal("a sync whose every object vanished under it reported success") + } + if listings := uploadListings(t, producer); len(listings) != 0 { + t.Fatalf("a sync that read none of its listed objects sent a listing %+v; it would tombstone the collection", listings) + } + }) + t.Run("object changed since its listing", func(t *testing.T) { + producer := &recordingProducer{} + svc, _ := newDatasourceService(newDatasourceFakeStore(), producer, nil) + fake := mirrorBucket() + fake.readErr = map[string]error{"loans/b.xlsx": objectstore.ErrObjectChanged} + useUploadBucket(svc, fake) + source := newUploadSource(t, svc) + if _, err := svc.RunDatasourceSync(context.Background(), source.ID); err == nil { + t.Fatal("a sync whose object changed under it reported success") } - listings := uploadListings(t, producer) - if len(listings) != 1 || len(*listings[0].Files) != 1 || (*listings[0].Files)[0].Key != "loans/a.xlsx" { - t.Fatalf("listing after a vanished object = %+v", listings) + if listings := uploadListings(t, producer); len(listings) != 0 { + t.Fatalf("a sync that could not read an object at its listed version sent a listing %+v", listings) } }) } diff --git a/module/services/accounts/code/pkg/business/datasource_upload_refusals_test.go b/module/services/accounts/code/pkg/business/datasource_upload_refusals_test.go new file mode 100644 index 000000000..548066736 --- /dev/null +++ b/module/services/accounts/code/pkg/business/datasource_upload_refusals_test.go @@ -0,0 +1,296 @@ +package business_test + +import ( + "context" + "errors" + "fmt" + "testing" + "time" + + storagev0 "github.com/codefly-dev/service-object-storage/gen/codefly/storage/v0" + "google.golang.org/genproto/googleapis/rpc/errdetails" + "google.golang.org/grpc" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" + "google.golang.org/protobuf/types/known/durationpb" + + "accounts/pkg/business" + "accounts/pkg/datasource/objectstore" + jobsv1 "accounts/pkg/gen/saas/jobs/v1" + "accounts/pkg/jobs" +) + +// listingService is the object-storage service as the host's real client +// reaches it: it serves a binding and a batched read, and answers List however +// the test scripts — including the way a regressed service would. +type listingService struct { + storagev0.ObjectStorageClient + list func(*storagev0.ListRequest) (*storagev0.ListResult, error) +} + +func (listingService) Capabilities(context.Context, *storagev0.CapabilitiesRequest, ...grpc.CallOption) (*storagev0.BackendCapabilities, error) { + return &storagev0.BackendCapabilities{MaxObjectBytes: 5 << 20, MaxListLimit: 1000, MaxGetManyItems: 1000}, nil +} + +func (l listingService) List(_ context.Context, r *storagev0.ListRequest, _ ...grpc.CallOption) (*storagev0.ListResult, error) { + return l.list(r) +} + +func runSyncJob(t *testing.T, svc *business.Service, producer *recordingProducer, source *business.DatasourceSource, jobSource string) error { + t.Helper() + if _, err := svc.SyncDatasourceSource(context.Background(), "actor-1", testOrg, source.ID); err != nil { + t.Fatal(err) + } + req := producer.jobs[len(producer.jobs)-1] + return svc.NewDatasourceSyncJobHandler()(context.Background(), &jobsv1.JobEnvelope{Queue: req.GetQueue(), Topic: req.GetTopic(), Source: jobSource, Attributes: req.GetAttributes()}) +} + +func processingFailure(t *testing.T, err error) *jobs.ProcessingError { + t.Helper() + var failure *jobs.ProcessingError + if !errors.As(err, &failure) { + t.Fatalf("job error = %v, want a classified ProcessingError", err) + } + return failure +} + +// TestUploadSyncNeverEmptiesASourceThatHasFiles is the host-owned invariant +// the owner set on review r2's g2: a sync never applies a deletion delta that +// empties a source. A listing that comes back empty for a source whose last +// closing snapshot named objects is refused with a named reason, closes +// nothing, and leaves the collection as it is — whatever the service answers. +// Driven through the host's real client and the production job handler, with +// the two service regressions the review injected: a rewritten scope that +// lists nothing, and a truncation discarded into an empty last page. Both +// answer OK with no objects and no continuation token. +func TestUploadSyncNeverEmptiesASourceThatHasFiles(t *testing.T) { + for _, regression := range []string{"rewritten scope", "discarded truncation"} { + t.Run(regression, func(t *testing.T) { + producer := &recordingProducer{} + svc, _ := newDatasourceService(newDatasourceFakeStore(), producer, nil) + useUploadBucket(svc, &fakeUploadBucket{objects: []fakeObject{{key: "/reports/a.md", etag: "a", body: "A"}}}) + cfg := uploadConfig() + cfg.Prefix = "/reports/" + source, err := svc.AddSource(context.Background(), "actor-1", business.AddSourceInput{ + OrgID: testOrg, Provider: business.DatasourceProviderUpload, CollectionLabel: "reports", Credential: "secretkey", Upload: cfg, + }) + if err != nil { + t.Fatal(err) + } + if _, err := svc.RunDatasourceSync(context.Background(), source.ID); err != nil { + t.Fatal(err) + } + if listings := uploadListings(t, producer); len(listings) != 1 || len(*listings[0].Files) != 1 { + t.Fatalf("the first sync's closing listing = %+v, want one file", listings) + } + + if err := svc.SetObjectStorage(context.Background(), listingService{list: func(*storagev0.ListRequest) (*storagev0.ListResult, error) { + return &storagev0.ListResult{}, nil + }}); err != nil { + t.Fatal(err) + } + _, err = svc.RunDatasourceSync(context.Background(), source.ID) + if !errors.Is(err, business.ErrUploadEmptyListingRefused) { + t.Fatalf("an empty listing for a source with files = %v, want ErrUploadEmptyListingRefused", err) + } + if listings := uploadListings(t, producer); len(listings) != 1 { + t.Fatalf("an empty listing for a source with files closed with %d listings; it must close none", len(listings)-1) + } + failure := processingFailure(t, runSyncJob(t, svc, producer, source, "github.sync.request")) + if failure.Retryable || failure.Failure.GetCode() != "datasource.upload_empty_listing_refused" { + t.Fatalf("job disposition = %+v, want the terminal empty-listing refusal", failure) + } + }) + } +} + +// TestUploadSyncClosesAnEmptyListingOnlyWhereNothingIsMirrored: the invariant +// refuses only what would empty a source with files. A source that never +// synced, or whose last snapshot named nothing, closes an empty listing as +// before; a source synced before the count was recorded is treated as having +// files. +func TestUploadSyncClosesAnEmptyListingOnlyWhereNothingIsMirrored(t *testing.T) { + producer := &recordingProducer{} + store := newDatasourceFakeStore() + svc, _ := newDatasourceService(store, producer, nil) + useUploadBucket(svc, &fakeUploadBucket{}) + add := func(label string) *business.DatasourceSource { + source, err := svc.AddExistingSource(context.Background(), "actor-1", business.AddSourceInput{ + OrgID: testOrg, Provider: business.DatasourceProviderUpload, CollectionLabel: label, Credential: "secretkey", Upload: uploadConfig(), + }) + if err != nil { + t.Fatal(err) + } + return source + } + fresh := add("fresh") + if _, err := svc.RunDatasourceSync(context.Background(), fresh.ID); err != nil { + t.Fatalf("an empty listing for a never-synced source = %v, want it closed", err) + } + if _, err := svc.RunDatasourceSync(context.Background(), fresh.ID); err != nil { + t.Fatalf("an empty listing for a source whose last snapshot was empty = %v, want it closed", err) + } + if listings := uploadListings(t, producer); len(listings) != 2 { + t.Fatalf("closing listings = %d, want both empty syncs closed", len(listings)) + } + + legacy := add("legacy") + synced := time.Now().Add(-time.Hour) + store.sources[legacy.ID].LastSyncedAt = &synced + if _, err := svc.RunDatasourceSync(context.Background(), legacy.ID); !errors.Is(err, business.ErrUploadEmptyListingRefused) { + t.Fatalf("an empty listing for a source synced before the count existed = %v, want it refused", err) + } +} + +// TestUploadRefusalsGetTheJobPolicyTheirReasonNames is review r2's g3: each +// reason the object-storage service names gets its own action at the job, not +// one generic retry. Driven through the host's real client and the production +// job handler. +func TestUploadRefusalsGetTheJobPolicyTheirReasonNames(t *testing.T) { + refusal := func(code codes.Code, reason string, details ...*errdetails.RetryInfo) error { + st, _ := status.New(code, "refused").WithDetails(&errdetails.ErrorInfo{Domain: "object-storage.codefly.dev", Reason: reason}) + for _, d := range details { + st, _ = st.WithDetails(d) + } + return st.Err() + } + for _, tc := range []struct { + reason string + err error + code string + retryable bool + degraded string // the reason the source is parked with, or "" when it is not + }{ + {"DESTINATION_REFUSED", refusal(codes.PermissionDenied, "DESTINATION_REFUSED"), "datasource.upload_destination_refused", false, business.ObjectStorageEndpointRefusedDegradeReason().String()}, + {"CREDENTIAL_REJECTED", refusal(codes.PermissionDenied, "CREDENTIAL_REJECTED"), "datasource.credential_rejected", false, business.ObjectStorageCredentialRejectedDegradeReason().String()}, + {"LISTING_INCOMPLETE", refusal(codes.Internal, "LISTING_INCOMPLETE"), "datasource.upload_listing_incomplete", false, ""}, + {"LISTING_TOO_LONG", refusal(codes.ResourceExhausted, "LISTING_TOO_LONG"), "datasource.upload_listing_incomplete", false, ""}, + {"RESPONSE_TOO_LARGE", refusal(codes.ResourceExhausted, "RESPONSE_TOO_LARGE"), "datasource.upload_listing_incomplete", false, ""}, + {"STORE_THROTTLED", refusal(codes.ResourceExhausted, "STORE_THROTTLED", &errdetails.RetryInfo{RetryDelay: durationpb.New(time.Minute)}), "datasource.rate_limited", true, ""}, + } { + t.Run(tc.reason, func(t *testing.T) { + producer := &recordingProducer{} + store := newDatasourceFakeStore() + svc, _ := newDatasourceService(store, producer, nil) + if err := svc.SetObjectStorage(context.Background(), listingService{list: func(*storagev0.ListRequest) (*storagev0.ListResult, error) { return nil, tc.err }}); err != nil { + t.Fatal(err) + } + source, err := svc.AddSource(context.Background(), "actor-1", business.AddSourceInput{ + OrgID: testOrg, Provider: business.DatasourceProviderUpload, CollectionLabel: "guides", Credential: "secretkey", Upload: uploadConfig(), + }) + if err != nil { + t.Fatal(err) + } + start := time.Now() + failure := processingFailure(t, runSyncJob(t, svc, producer, source, "github.sync.request")) + if failure.Failure.GetCode() != tc.code || failure.Retryable != tc.retryable { + t.Fatalf("disposition = %s retryable=%v, want %s retryable=%v", failure.Failure.GetCode(), failure.Retryable, tc.code, tc.retryable) + } + if tc.retryable && !failure.NotBefore.After(start.Add(30*time.Second)) { + t.Fatalf("a throttle retries at %v, want no earlier than the store's reset", failure.NotBefore) + } + got := store.sources[source.ID] + if tc.degraded != "" && (got.Status != business.DatasourceStatusDegraded || got.StatusReason != tc.degraded) { + t.Fatalf("source = %s %q, want degraded with %q", got.Status, got.StatusReason, tc.degraded) + } + if tc.degraded == "" && got.Status == business.DatasourceStatusDegraded { + t.Fatalf("a %s refusal parked the source", tc.reason) + } + if listings := uploadListings(t, producer); len(listings) != 0 { + t.Fatalf("a refused sync closed with %d listings", len(listings)) + } + }) + } +} + +// degradeFailingStore is a store whose park write fails while broken is set. +type degradeFailingStore struct { + *datasourceFakeStore + broken bool +} + +func (s *degradeFailingStore) MarkDatasourceSourceDegraded(ctx context.Context, id string, reason business.DatasourceDegradeReason) error { + if s.broken { + return errors.New("store unavailable") + } + return s.datasourceFakeStore.MarkDatasourceSourceDegraded(ctx, id, reason) +} + +// TestUploadRefusalWhoseParkFailsRetries: the park is what makes a refused +// source's stop visible, so a refusal whose park write fails is not terminal — +// a terminal job would leave the source reading active with nothing on it to +// say why it stopped. The job retries instead, and the next attempt parks it. +// The rule the GitHub public-source park already followed; both now share it. +func TestUploadRefusalWhoseParkFailsRetries(t *testing.T) { + refused, _ := status.New(codes.PermissionDenied, "refused").WithDetails(&errdetails.ErrorInfo{Domain: "object-storage.codefly.dev", Reason: "DESTINATION_REFUSED"}) + producer := &recordingProducer{} + store := °radeFailingStore{datasourceFakeStore: newDatasourceFakeStore(), broken: true} + svc, _ := newDatasourceService(store, producer, nil) + if err := svc.SetObjectStorage(context.Background(), listingService{list: func(*storagev0.ListRequest) (*storagev0.ListResult, error) { return nil, refused.Err() }}); err != nil { + t.Fatal(err) + } + source, err := svc.AddSource(context.Background(), "actor-1", business.AddSourceInput{ + OrgID: testOrg, Provider: business.DatasourceProviderUpload, CollectionLabel: "guides", Credential: "secretkey", Upload: uploadConfig(), + }) + if err != nil { + t.Fatal(err) + } + + err = runSyncJob(t, svc, producer, source, "github.sync.request") + var failure *jobs.ProcessingError + if err == nil || (errors.As(err, &failure) && !failure.Retryable) { + t.Fatalf("a refusal whose park failed = %v, want a retryable failure", err) + } + if errors.Is(err, objectstore.ErrDestinationRefused) { + t.Fatalf("a refusal whose park failed still carries the refusal: %v", err) + } + if got := store.sources[source.ID]; got.Status == business.DatasourceStatusDegraded { + t.Fatalf("source = %s, want it as it was while the park cannot be written", got.Status) + } + + store.broken = false + if failure := processingFailure(t, runSyncJob(t, svc, producer, source, "github.sync.request")); failure.Retryable || failure.Failure.GetCode() != "datasource.upload_destination_refused" { + t.Fatalf("retry = %s retryable=%v, want the terminal refusal once parked", failure.Failure.GetCode(), failure.Retryable) + } + if got := store.sources[source.ID]; got.Status != business.DatasourceStatusDegraded || got.StatusReason != business.ObjectStorageEndpointRefusedDegradeReason().String() { + t.Fatalf("source = %s %q, want parked by the retry", got.Status, got.StatusReason) + } +} + +// TestUploadSyncSkipsAnObjectTheServiceCapsAndContinues: an object the +// object-storage service refuses as over its cap is skipped and the sync goes +// on — the rest of its batch and every later batch are delivered — then +// surfaces it without a closing listing that would tombstone it. Seventy-five +// objects listed at 900 KiB make two batches (a batch is bounded at 64 MiB); +// the capped one is in the first. +func TestUploadSyncSkipsAnObjectTheServiceCapsAndContinues(t *testing.T) { + producer := &recordingProducer{} + svc, _ := newDatasourceService(newDatasourceFakeStore(), producer, nil) + var objects []fakeObject + for i := 0; i < 75; i++ { + objects = append(objects, fakeObject{key: fmt.Sprintf("kb/%04d.md", i), etag: fmt.Sprint(i), body: "x", size: 900 << 10}) + } + useUploadBucket(svc, &fakeUploadBucket{ + objects: objects, + readErr: map[string]error{"kb/0003.md": &objectstore.TooLargeError{Keys: []string{"kb/0003.md"}}}, + skipOnErr: true, + }) + cfg := uploadConfig() + source, err := svc.AddExistingSource(context.Background(), "actor-1", business.AddSourceInput{ + OrgID: testOrg, Provider: business.DatasourceProviderUpload, CollectionLabel: "guides", Credential: "secretkey", Upload: cfg, + }) + if err != nil { + t.Fatal(err) + } + takeConnectSync(t, producer, source) + enqueued, err := svc.RunDatasourceSync(context.Background(), source.ID) + if err == nil { + t.Fatal("a sync with a skipped object reported a clean success") + } + if enqueued != 74 { + t.Fatalf("delivered %d objects, want every object but the capped one, across both batches", enqueued) + } + if listings := uploadListings(t, producer); len(listings) != 0 { + t.Fatalf("a sync that skipped an object closed with a listing; it would tombstone it") + } +} diff --git a/module/services/accounts/code/pkg/business/datasources.go b/module/services/accounts/code/pkg/business/datasources.go index fc692103e..92c2fb7d3 100644 --- a/module/services/accounts/code/pkg/business/datasources.go +++ b/module/services/accounts/code/pkg/business/datasources.go @@ -23,6 +23,7 @@ import ( "accounts/pkg/jobs" "github.com/codefly-dev/core/wool" + storagev0 "github.com/codefly-dev/service-object-storage/gen/codefly/storage/v0" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" ) @@ -81,6 +82,19 @@ func SnapshotTooLargeDegradeReason(size, limit int) DatasourceDegradeReason { // stated as fact. const DatasourceReasonPublicRepositoryUnreadable = "GitHub no longer serves this repository without a credential — it may have been made private, renamed or deleted. Reconnect it through the GitHub App or with a repository-scoped fine-grained PAT." +// ObjectStorageEndpointRefusedDegradeReason explains an object-storage source +// whose endpoint the object-storage service refuses to dial: not on the public +// internet, nor a composed dependency. Fixed text, nothing provider-supplied. +func ObjectStorageEndpointRefusedDegradeReason() DatasourceDegradeReason { + return DatasourceDegradeReason{text: "The object-storage service refuses this source's endpoint: it is not on the public internet. Connect the source with a public endpoint to resume syncing."} +} + +// ObjectStorageCredentialRejectedDegradeReason explains an object-storage +// source whose store rejected its access key. Fixed text. +func ObjectStorageCredentialRejectedDegradeReason() DatasourceDegradeReason { + return DatasourceDegradeReason{text: "The store rejected this source's access key. Connect the source again with a valid key to resume syncing."} +} + // PublicRepositoryUnreadableDegradeReason is that sentence as a stored reason. func PublicRepositoryUnreadableDegradeReason() DatasourceDegradeReason { return DatasourceDegradeReason{text: DatasourceReasonPublicRepositoryUnreadable} @@ -365,6 +379,11 @@ type UploadDatasourceConfig struct { Prefix string `json:"prefix,omitempty"` AccessKeyID string `json:"access_key_id"` MaxObjects int `json:"max_objects,omitempty"` + // Which keys under the prefix are ingested: RE2 patterns over the key with + // the prefix removed, and a file-type allowlist (uploadKeyScope). + IncludePatterns []string `json:"include_patterns,omitempty"` + ExcludePatterns []string `json:"exclude_patterns,omitempty"` + FileExtensions []string `json:"file_extensions,omitempty"` } // DatasourceSource is one connected external datasource. CredentialSecretRef and @@ -509,14 +528,6 @@ type CrawlerContentClient interface { Fetch(ctx context.Context, pageURL string) (crawler.Page, error) } -// UploadContentClient is the subset of the object-storage connector the Service -// needs. The Service builds one per Source from its config and decrypted secret -// access key and drives it item-at-a-time (List then Fetch). -type UploadContentClient interface { - List(ctx context.Context) (objectstore.Listing, error) - Fetch(ctx context.Context, key string) (objectstore.Object, error) -} - // OAuth2RefreshFunc exchanges an OAuth 2.0 refresh token for a fresh access // token. It matches apisource.RefreshOAuth2; tests substitute a deterministic // implementation. @@ -552,18 +563,6 @@ func (s *Service) SetDatasourceConnector(cipher SecretCipher, producer jobs.Prod return crawler.New(crawler.Config{SitemapURL: cfg.SitemapURL, MaxPages: cfg.MaxPages}) } } - if s.newUploadClient == nil { - s.newUploadClient = func(cfg UploadDatasourceConfig, secretAccessKey string) UploadContentClient { - return objectstore.New(objectstore.Config{ - Endpoint: cfg.Endpoint, - Region: cfg.Region, - Bucket: cfg.Bucket, - Prefix: cfg.Prefix, - AccessKeyID: cfg.AccessKeyID, - MaxObjects: cfg.MaxObjects, - }, secretAccessKey) - } - } if s.newOAuth2Refresh == nil { s.newOAuth2Refresh = apisource.RefreshOAuth2 } @@ -617,10 +616,27 @@ func (s *Service) SetDatasourceCrawlerClientFactory(factory func(cfg CrawlerData s.newCrawlerClient = factory } -// SetDatasourceUploadClientFactory overrides how per-Source object-storage -// clients are built. Tests use it to inject a fake without a live object store. -func (s *Service) SetDatasourceUploadClientFactory(factory func(cfg UploadDatasourceConfig, secretAccessKey string) UploadContentClient) { - s.newUploadClient = factory +// SetObjectStorage makes the composed object-storage service the store every +// object-storage source is read through, under a per-call binding of the +// source's bucket and credential — if, and only if, the service can serve one: +// it asks first (objectstore.CheckService). A service that cannot — a release +// from before the binding, such as 0.0.18 — is refused and left unwired, so +// object storage is unavailable: the catalog does not offer it, AddSource +// refuses it, and nothing is admitted that every sync would then fail on. +func (s *Service) SetObjectStorage(ctx context.Context, client storagev0.ObjectStorageClient) error { + if err := objectstore.CheckService(ctx, client); err != nil { + return err + } + s.newUploadBucket = func(cfg objectstore.SourceConfig, accessKeyID, secretAccessKey string) objectstore.Bucket { + return objectstore.NewServiceBucket(client, cfg, accessKeyID, secretAccessKey) + } + return nil +} + +// SetDatasourceUploadBucketFactory overrides how an object-storage source's +// Bucket is built. Tests use it to script a store without the service. +func (s *Service) SetDatasourceUploadBucketFactory(factory func(cfg objectstore.SourceConfig, accessKeyID, secretAccessKey string) objectstore.Bucket) { + s.newUploadBucket = factory } // SetDatasourceOAuth2RefreshFunc overrides how an OAuth 2.0 access token is @@ -934,6 +950,10 @@ func (s *Service) addSource(ctx context.Context, actorID string, input AddSource } cfg, err := normalizeUploadConfig(input.Upload) if err != nil { + // A key-filter refusal names its field; it reaches the caller as itself. + if status.Code(err) == codes.InvalidArgument { + return nil, err + } return nil, w.Wrap(err) } source.Upload = cfg @@ -1117,6 +1137,22 @@ func normalizeCrawlerConfig(cfg *CrawlerDatasourceConfig) (*CrawlerDatasourceCon return &CrawlerDatasourceConfig{SitemapURL: sitemapURL, MaxPages: maxPages}, nil } +// normalizeUploadKeyFilter validates from's key filter into to, naming each +// field under path (its place in the request) when it refuses one. +func normalizeUploadKeyFilter(to, from *UploadDatasourceConfig, path string) error { + var err error + if to.IncludePatterns, err = normalizeKeyPatterns(path+"include_patterns", from.IncludePatterns); err != nil { + return err + } + if to.ExcludePatterns, err = normalizeKeyPatterns(path+"exclude_patterns", from.ExcludePatterns); err != nil { + return err + } + if to.FileExtensions, err = normalizeFileExtensions(from.FileExtensions); err != nil { + return keyFilterRefused(path+"file_extensions", status.Convert(err).Message()) + } + return nil +} + // normalizeUploadConfig validates and trims an object-storage provider config. func normalizeUploadConfig(cfg *UploadDatasourceConfig) (*UploadDatasourceConfig, error) { if cfg == nil { @@ -1127,8 +1163,13 @@ func normalizeUploadConfig(cfg *UploadDatasourceConfig) (*UploadDatasourceConfig if err != nil || (parsed.Scheme != "http" && parsed.Scheme != "https") || parsed.Host == "" { return nil, errors.New("upload endpoint must be an absolute http(s) url") } + // The object-storage service binds a store by its origin alone; a path, + // query or userinfo would be refused at every sync, so it is refused here. + if parsed.User != nil || strings.Trim(parsed.Path, "/") != "" || parsed.RawQuery != "" || parsed.Fragment != "" { + return nil, errors.New("upload endpoint must be the store's origin, with no path, query or userinfo") + } out := &UploadDatasourceConfig{ - Endpoint: endpoint, + Endpoint: parsed.Scheme + "://" + parsed.Host, Region: strings.TrimSpace(cfg.Region), Bucket: strings.TrimSpace(cfg.Bucket), Prefix: strings.TrimSpace(cfg.Prefix), @@ -1147,6 +1188,9 @@ func normalizeUploadConfig(cfg *UploadDatasourceConfig) (*UploadDatasourceConfig if out.MaxObjects < 0 { return nil, errors.New("upload max objects must not be negative") } + if err := normalizeUploadKeyFilter(out, cfg, "upload."); err != nil { + return nil, err + } return out, nil } @@ -1883,103 +1927,185 @@ func crawlerIngestIdempotencyKey(sourceID, pageURL, contentSHA string) string { } // ErrUploadListingIncomplete reports an upload sync whose listing was not the -// whole folder: the store truncated it, or the source's object cap cut it. The -// objects it did list are still delivered, but no listing is sent, so nothing -// is removed on a partial view; it fails the sync terminally rather than -// retrying into the same cap. +// whole scope: the prefix runs past what one listing pages through, or more +// objects pass the key filter than the source's object cap. Nothing is +// delivered and no listing is sent, so nothing is removed on a partial view; it +// fails the sync terminally rather than retrying into the same cap. var ErrUploadListingIncomplete = errors.New("upload listing is incomplete; removals were not reconciled") -// runUploadSync mirrors the source's folder into the documents store. It lists -// the source's objects and fetches each one at a time, enqueuing an ingest -// delivery per object, then closes the sync with the complete listing of every -// object it delivered, so the store tombstones the entries whose objects are -// gone. Every job of one sync carries one freshly allocated ordinal, which the -// store ranks the listing against the writes it covers by. +// runUploadSync mirrors the source's scope into the documents store. It takes a +// complete snapshot through the object-storage connector — the listing within +// the source's key filter and object cap, read through the object-storage +// service — and fetches it in batches, enqueuing an ingest delivery per +// object, then closes the sync with the complete listing of every object it +// delivered, so the store tombstones the entries whose objects are gone. Every +// job of one sync carries one freshly allocated ordinal, which the store ranks +// the listing against the writes it covers by. // -// The listing is sent only when the sync saw the whole folder and delivered -// every object in it: a failed or truncated listing, a fetch failure, or an -// object too large to deliver ends the sync without one, so a partial view can -// never delete. A listed object deleted before its fetch is gone, and is left -// out of the listing. A key ending in "/" is a folder marker, not a file, and -// is neither delivered nor listed. +// The listing is sent only when the sync saw the whole scope and delivered +// every object in it: a failed listing, one the cap cuts, a fetch failure, or +// an object too large to deliver ends the sync without one, so a partial view +// can never delete. An object rewritten or deleted between the listing and its +// read fails the sync, which the next one lists afresh. func (s *Service) runUploadSync(ctx context.Context, source *DatasourceSource) (int, error) { w := wool.Get(ctx).In("runUploadSync") - if s.newUploadClient == nil { + // The registered connector is the metered one: every listing and batched + // read spends the source credential's budget, as a GitHub sync does. + conn, ok := s.datasourceConnectors.Files(DatasourceProviderUpload) + if !ok { return 0, w.NewError("datasource connector is not configured") } if source.Upload == nil { return 0, w.NewError("upload source has no config") } - - secretKey, err := s.datasourceCipher.DecryptSecret(ctx, DatasourceConnectorSecretPurpose(source.ID), source.CredentialSecretRef) - if err != nil { - return 0, w.Wrapf(err, "decrypt secret access key") + src := connectorSource(source) + snapshot, err := conn.Changes(ctx, src, "") + if errors.Is(err, objectstore.ErrListingIncomplete) { + return 0, w.Wrapf(fmt.Errorf("%w: %w", ErrUploadListingIncomplete, err), "list objects under %q", source.Upload.Prefix) } - client := s.newUploadClient(*source.Upload, secretKey) - - listing, err := client.List(ctx) if err != nil { - return 0, w.Wrapf(err, "list objects") + return 0, w.Wrapf(s.parkRefusedUploadSource(ctx, source, err), "list objects") + } + // A host-owned invariant, whatever any service version answers: a sync + // never closes with a snapshot that empties a source that has files. Such a + // snapshot tombstones every entry mirrored from the source, and a store or + // service that wrongly answers "empty" — a rewritten scope, a discarded + // truncation — is the case that has already happened. Emptying a source is + // an explicit operation, not a side effect of a listing. + if len(snapshot.Changes) == 0 { + held, err := s.uploadSourceMayHoldFiles(ctx, source) + if err != nil { + return 0, w.Wrapf(err, "read the source's last snapshot") + } + if held { + return 0, w.Wrapf(ErrUploadEmptyListingRefused, "listing under %q", source.Upload.Prefix) + } } ordinal, err := s.allocateOrdinal(ctx, source.ID) if err != nil { return 0, w.Wrapf(err, "allocate ordinal") } - enqueued := 0 - oversized := 0 - delivered := make([]uploadListingFile, 0, len(listing.Entries)) - for _, entry := range listing.Entries { - if strings.HasSuffix(entry.Key, "/") { - continue + enqueued, oversized := 0, 0 + delivered := make([]uploadListingFile, 0, len(snapshot.Changes)) + var batch []connector.FileRef + var batchBytes int64 + flush := func() error { + if len(batch) == 0 { + return nil } - object, err := client.Fetch(ctx, entry.Key) - if err != nil { - if errors.Is(err, objectstore.ErrObjectNotFound) { - // Deleted between listing and fetch: it is gone, so it is not listed. - w.Warn("skipping vanished object", wool.Field("key", entry.Key)) - continue + err := conn.FetchFiles(ctx, src, snapshot.To, batch, func(f connector.File, content io.Reader) error { + body, err := io.ReadAll(content) + if err != nil { + return err } - if errors.Is(err, objectstore.ErrObjectTooLarge) { - oversized++ - continue + fingerprint := uploadFingerprint(f.Provenance.GetItemVersion(), body) + if err := s.enqueueUploadIngest(ctx, source, f.Locator, body, f.ContentType, fingerprint, ordinal); err != nil { + return w.Wrapf(err, "enqueue %s", f.Locator) } - // A permission or transport failure must not read as an empty bucket: - // surface it so the sync is retried and the misconfiguration is visible. - return enqueued, w.Wrapf(err, "fetch %s", entry.Key) + delivered = append(delivered, uploadListingFile{Key: f.Locator, ETag: fingerprint}) + enqueued++ + return nil + }) + batch, batchBytes = batch[:0], 0 + // An object the object-storage service refuses as over its cap is + // skipped and the rest of the batch is delivered; like an object over + // the ingest limit, it keeps this sync from sending a listing that + // would tombstone it. + var skipped *objectstore.TooLargeError + if errors.As(err, &skipped) { + oversized += len(skipped.Keys) + return nil } - if len(object.Body) > maxIngestPayload { + if err != nil { + return s.parkRefusedUploadSource(ctx, source, err) + } + return nil + } + for _, ch := range snapshot.Changes { + // Objects too large for the ingest inbox cannot be delivered and the + // consumer holds no credentials to re-fetch them: never read, counted. + if ch.Size > maxIngestPayload { oversized++ continue } - fingerprint := uploadFingerprint(object, entry) - if err := s.enqueueUploadIngest(ctx, source, object, fingerprint, ordinal); err != nil { - return enqueued, w.Wrapf(err, "enqueue %s", entry.Key) + if len(batch) == objectstore.MaxObjectsPerBatch || batchBytes+ch.Size > objectstore.MaxBatchBytes { + if err := flush(); err != nil { + return enqueued, w.Wrapf(err, "fetch objects") + } } - delivered = append(delivered, uploadListingFile{Key: object.Key, ETag: fingerprint}) - enqueued++ + batch = append(batch, connector.FileRef{ItemID: ch.Locator, ItemVersion: ch.ItemVersion}) + batchBytes += ch.Size + } + if err := flush(); err != nil { + return enqueued, w.Wrapf(err, "fetch objects") } - // Objects too large for the ingest inbox cannot be delivered and the consumer - // holds no credentials to re-fetch them; surface it instead of a clean sync. if oversized > 0 { return enqueued, w.NewError("%d object(s) exceed the ingest payload limit", oversized) } - if !listing.Complete { - return enqueued, w.Wrapf(ErrUploadListingIncomplete, "listing of %d object(s) under %q", len(listing.Entries), source.Upload.Prefix) - } if err := s.enqueueUploadListing(ctx, source, ordinal, delivered); err != nil { return enqueued, w.Wrapf(err, "enqueue listing") } if err := s.store.WithOrgTx(ctx, source.OrgID, func(ctx context.Context) error { - return s.store.SetDatasourceSourceSynced(ctx, source.OrgID, source.ID, time.Now().UTC()) + return s.store.RecordDatasourceSnapshotSent(ctx, source.OrgID, source.ID, time.Now().UTC(), len(delivered)) }); err != nil { return enqueued, w.Wrapf(err, "record sync time") } return enqueued, nil } +// ErrUploadEmptyListingRefused reports an object-storage sync whose listing +// came back empty for a source that has files. It is refused, not closed: the +// collection is left as it is, and emptying a source is an explicit operation. +var ErrUploadEmptyListingRefused = errors.New("upload listing is empty for a source that has files; refused rather than tombstoning every entry") + +// uploadSourceMayHoldFiles reports whether an object-storage source's mirrored +// entries may be non-empty: its last closing snapshot named objects, or it +// synced before that count was recorded (migration 60), which is unknown and +// so treated as having files. A source that never synced holds nothing. +func (s *Service) uploadSourceMayHoldFiles(ctx context.Context, source *DatasourceSource) (bool, error) { + var items int + var known bool + err := s.store.WithOrgTx(ctx, source.OrgID, func(ctx context.Context) error { + var err error + items, known, err = s.store.DatasourceSourceLastSnapshot(ctx, source.OrgID, source.ID) + return err + }) + if err != nil { + return false, err + } + if known { + return items > 0, nil + } + return source.LastSyncedAt != nil, nil +} + +// parkRefusedUploadSource parks an object-storage source a retry cannot fix: +// the object-storage service's guard refused its endpoint, or its store +// rejected its access key. The source is degraded with a reason a person can +// act on — change the endpoint, or connect it again with a valid key — so the +// reconcile sweep stops retrying it. Any other error is returned as it is. A +// park that fails returns its own error in place of the refusal, so the job +// retries and the park is attempted again (parkDatasourceSource). +func (s *Service) parkRefusedUploadSource(ctx context.Context, source *DatasourceSource, err error) error { + var reason DatasourceDegradeReason + switch { + case errors.Is(err, objectstore.ErrDestinationRefused): + reason = ObjectStorageEndpointRefusedDegradeReason() + case errors.Is(err, connector.ErrCredentialRefused): + reason = ObjectStorageCredentialRejectedDegradeReason() + default: + return err + } + if _, perr := s.parkDatasourceSource(ctx, source, reason); perr != nil { + wool.Get(ctx).Warn("could not park a refused object-storage source", wool.Field("source_id", source.ID), wool.ErrField(perr)) + return fmt.Errorf("park the refused object-storage source: %w", perr) + } + return err +} + // uploadListingFile is one delivered object in an upload sync's listing: its // key and the fingerprint its own ingest job was versioned by. type uploadListingFile struct { @@ -2034,24 +2160,20 @@ func (s *Service) enqueueUploadListing(ctx context.Context, source *DatasourceSo } // uploadFingerprint is the object's content fingerprint used to key delivery -// idempotency: the ETag from the GET response, else the ETag from the listing, -// else a hash of the body — so a re-pull of unchanged content always dedupes. -func uploadFingerprint(object objectstore.Object, entry objectstore.Entry) string { - if object.ETag != "" { - return object.ETag +// idempotency: the ETag the object was listed and read at, else a hash of the +// body — so a re-pull of unchanged content always dedupes. +func uploadFingerprint(etag string, body []byte) string { + if etag != "" { + return etag } - if entry.ETag != "" { - return entry.ETag - } - digest := sha256.Sum256(object.Body) + digest := sha256.Sum256(body) return hex.EncodeToString(digest[:]) } -func (s *Service) enqueueUploadIngest(ctx context.Context, source *DatasourceSource, object objectstore.Object, fingerprint string, ordinal int64) error { +func (s *Service) enqueueUploadIngest(ctx context.Context, source *DatasourceSource, key string, body []byte, contentType, fingerprint string, ordinal int64) error { if source.BoundaryNodeID == "" { return wool.Get(ctx).NewError("datasource source has no resolvable boundary") } - contentType := object.ContentType if contentType == "" { contentType = datasourceIngestContentType } @@ -2062,9 +2184,9 @@ func (s *Service) enqueueUploadIngest(ctx context.Context, source *DatasourceSou Queue: DatasourceIngestQueue, Topic: datasourceUploadSyncTopic, Source: datasourceUploadSyncSource, - IdempotencyKey: uploadIngestIdempotencyKey(source.ID, object.Key, fingerprint+"\x00"+strconv.FormatInt(ordinal, 10)), + IdempotencyKey: uploadIngestIdempotencyKey(source.ID, key, fingerprint+"\x00"+strconv.FormatInt(ordinal, 10)), SchemaVersion: datasourceIngestSchemaVersion, - Payload: object.Body, + Payload: body, ContentType: contentType, MaxAttempts: datasourceIngestMaxAttempts, Attributes: map[string]string{ @@ -2072,7 +2194,7 @@ func (s *Service) enqueueUploadIngest(ctx context.Context, source *DatasourceSou attrOrgID: source.OrgID, attrBoundaryID: source.BoundaryNodeID, attrUploadBucket: source.Upload.Bucket, - attrUploadKey: object.Key, + attrUploadKey: key, attrUploadETag: fingerprint, attrUploadOrdinal: strconv.FormatInt(ordinal, 10), attrChangeType: changeTypeAdded, @@ -2108,6 +2230,12 @@ func (s *Service) NewDatasourceSyncJobHandler() jobs.Handler { if sourceID == "" { return jobs.NewProcessingError("datasource.invalid_job", "datasource sync job has no source id", false) } + // A sync a person asked for ("Sync now", the first sync at connect) is + // interactive and may spend the credential's reserved share; the + // scheduled pass is background and yields — as a GitHub delivery does. + if envelope.GetSource() == datasourceSyncRequestSource { + ctx = connector.WithPriority(ctx, connector.PriorityInteractive) + } _, err := s.RunDatasourceSync(ctx, sourceID) switch { case err == nil, errors.Is(err, ErrDatasourceSourceNotFound): @@ -2116,13 +2244,32 @@ func (s *Service) NewDatasourceSyncJobHandler() jobs.Handler { // The folder is larger than one listing reaches; a retry lists the // same page. Fail terminally so it is visible, not retried forever. return jobs.NewProcessingError("datasource.upload_listing_incomplete", err.Error(), false) + case errors.Is(err, ErrUploadEmptyListingRefused): + // The collection is left as it is; a retry would list the same. + return jobs.NewProcessingError("datasource.upload_empty_listing_refused", ErrUploadEmptyListingRefused.Error(), false) + case errors.Is(err, objectstore.ErrDestinationRefused): + // The source is parked: its endpoint must change, not be retried. + return jobs.NewProcessingError("datasource.upload_destination_refused", "the object-storage service refuses the source's endpoint", false) + case errors.Is(err, connector.ErrCredentialRefused): + // The source is parked until it is connected with a valid key. + return jobs.NewProcessingError("datasource.credential_rejected", "the store rejected the source's credential", false) case errors.Is(err, ErrOAuth2ReauthRequired): // The refresh token is permanently dead; replaying it can never // succeed, so fail the job terminally rather than burning retries. return jobs.NewProcessingError("datasource.oauth2_reauth_required", err.Error(), false) - default: - return err } + // A throttle — the store's, or the host's own budget yielding — backs + // off until the limit resets rather than on the queue's schedule. + var limited *connector.RateLimitedError + if errors.As(err, &limited) { + failure := jobs.NewProcessingError("datasource.rate_limited", "the provider rate limited the source's credential", true) + var typed *jobs.ProcessingError + if errors.As(failure, &typed) { + typed.NotBefore = limited.ResetAt + } + return failure + } + return err } } diff --git a/module/services/accounts/code/pkg/business/datasources_test.go b/module/services/accounts/code/pkg/business/datasources_test.go index 9be154ded..c5528638d 100644 --- a/module/services/accounts/code/pkg/business/datasources_test.go +++ b/module/services/accounts/code/pkg/business/datasources_test.go @@ -36,8 +36,10 @@ type datasourceFakeStore struct { // label alongside the row. Modelling existence alone let a source come back // with an empty collection name and no test could see it. nodes map[string]string - collections map[string]string // label -> node id - ordinals map[string]int64 // source id -> next ordinal to hand out + collections map[string]string // label -> node id + ordinals map[string]int64 // source id -> next ordinal to hand out + manifests map[string][]objectstore.Item // org/source/version -> retained listing + snapshots map[string]int // source id -> objects its last closing snapshot named // beforeInstallationMark, when set, runs just before a park is applied. beforeInstallationMark func(sourceID string) @@ -60,6 +62,8 @@ func newDatasourceFakeStore() *datasourceFakeStore { nodes: map[string]string{}, collections: map[string]string{}, ordinals: map[string]int64{}, + manifests: map[string][]objectstore.Item{}, + snapshots: map[string]int{}, setups: map[string]*business.GitHubAppSetup{}, setupConsumed: map[string]bool{}, @@ -143,7 +147,7 @@ func (f *datasourceFakeStore) InsertDatasourceSource(_ context.Context, source * } // UpdateDatasourceSourceFetchScope mirrors the store's narrow UPDATE: only the -// three fetch-scope fields are written, so a business change that tried to move +// fetch-scope fields and the config envelope are written, so a business change that tried to move // a source's repository or credential through this call fails here as it would // against the statement. // @@ -162,6 +166,12 @@ func (f *datasourceFakeStore) UpdateDatasourceSourceFetchScope(_ context.Context stored.Branch = source.Branch stored.Paths = source.Paths stored.FileExtensions = source.FileExtensions + // The statement also rewrites the config envelope, which for an + // object-storage source is its whole config, key filter included. + if source.Upload != nil { + cfg := *source.Upload + stored.Upload = &cfg + } return nil } @@ -285,6 +295,37 @@ func (f *datasourceFakeStore) DeleteDatasourceSource(_ context.Context, orgID, i }, nil } +func (f *datasourceFakeStore) PutDatasourceObjectManifest(_ context.Context, orgID, sourceID, version string, objects []objectstore.Item) error { + f.mu.Lock() + defer f.mu.Unlock() + f.manifests[orgID+"/"+sourceID+"/"+version] = append([]objectstore.Item(nil), objects...) + return nil +} + +func (f *datasourceFakeStore) GetDatasourceObjectManifest(_ context.Context, orgID, sourceID, version string) ([]objectstore.Item, bool, error) { + f.mu.Lock() + defer f.mu.Unlock() + items, ok := f.manifests[orgID+"/"+sourceID+"/"+version] + return items, ok, nil +} + +func (f *datasourceFakeStore) RecordDatasourceSnapshotSent(ctx context.Context, orgID, id string, syncedAt time.Time, items int) error { + if err := f.SetDatasourceSourceSynced(ctx, orgID, id, syncedAt); err != nil { + return err + } + f.mu.Lock() + defer f.mu.Unlock() + f.snapshots[id] = items + return nil +} + +func (f *datasourceFakeStore) DatasourceSourceLastSnapshot(_ context.Context, _ string, id string) (int, bool, error) { + f.mu.Lock() + defer f.mu.Unlock() + items, ok := f.snapshots[id] + return items, ok, nil +} + func (f *datasourceFakeStore) SetDatasourceSourceSynced(_ context.Context, orgID, id string, syncedAt time.Time) error { f.mu.Lock() defer f.mu.Unlock() @@ -1265,33 +1306,71 @@ func (f *fakeCrawlerClient) Fetch(_ context.Context, pageURL string) (crawler.Pa return crawler.Page{}, errors.New("unknown page") } -// fakeUploadClient serves fixed objects without a live object store, driven one -// key at a time. fetchErr injects a per-key error (e.g. objectstore.ErrObjectNotFound, -// objectstore.ErrObjectTooLarge, or a generic failure); listErr fails the listing. -type fakeUploadClient struct { - entries []objectstore.Entry - objects map[string]objectstore.Object - fetchErr map[string]error - listErr error - incomplete bool - listCalls int - fetchCalls int +// fakeUploadBucket is a scripted store behind the object-storage connector: it +// lists fixed objects and reads a batch in one Read call. readErr injects a +// per-key read error (an object changed since its listing, a refused +// credential); listErr fails the listing. +type fakeUploadBucket struct { + objects []fakeObject + contentType string + readErr map[string]error + skipOnErr bool // a readErr item is skipped and the read continues, as ServiceBucket does for an object over the service's cap + listErr error + listCalls int + readCalls int + read []string } -func (f *fakeUploadClient) List(context.Context) (objectstore.Listing, error) { +// fakeObject is one listed object; size overrides len(body) when set. +type fakeObject struct { + key, etag, body string + size int64 +} + +func (f *fakeUploadBucket) List(context.Context) ([]objectstore.Item, error) { f.listCalls++ if f.listErr != nil { - return objectstore.Listing{}, f.listErr + return nil, f.listErr + } + items := make([]objectstore.Item, 0, len(f.objects)) + for _, o := range f.objects { + size := o.size + if size == 0 { + size = int64(len(o.body)) + } + items = append(items, objectstore.Item{Key: o.key, ETag: o.etag, Size: size}) } - return objectstore.Listing{Entries: f.entries, Complete: !f.incomplete}, nil + return items, nil } -func (f *fakeUploadClient) Fetch(_ context.Context, key string) (objectstore.Object, error) { - f.fetchCalls++ - if err := f.fetchErr[key]; err != nil { - return objectstore.Object{}, err +func (f *fakeUploadBucket) Read(_ context.Context, items []objectstore.Item, visit func(objectstore.Item, string, io.Reader) error) error { + f.readCalls++ + var skipped error + for _, it := range items { + f.read = append(f.read, it.Key) + if err := f.readErr[it.Key]; err != nil { + if f.skipOnErr { + skipped = err + continue + } + return err + } + body := "" + for _, o := range f.objects { + if o.key == it.Key { + body = o.body + } + } + if err := visit(it, f.contentType, strings.NewReader(body)); err != nil { + return err + } } - return f.objects[key], nil + return skipped +} + +// useUploadBucket makes fake the store every object-storage source reads. +func useUploadBucket(svc *business.Service, fake *fakeUploadBucket) { + svc.SetDatasourceUploadBucketFactory(func(objectstore.SourceConfig, string, string) objectstore.Bucket { return fake }) } func crawlerConfig() *business.CrawlerDatasourceConfig { @@ -1525,16 +1604,13 @@ func TestRunDatasourceSync_CrawlerWholesaleFailureIsNotEmptySuccess(t *testing.T func TestRunDatasourceSync_UploadStreamsPerObject(t *testing.T) { producer := &recordingProducer{} svc, _ := newDatasourceService(newDatasourceFakeStore(), producer, nil) - var gotSecret string - fake := &fakeUploadClient{ - entries: []objectstore.Entry{{Key: "kb/a.pdf", ETag: "etag-a"}, {Key: "kb/b.pdf", ETag: "etag-b"}}, - objects: map[string]objectstore.Object{ - "kb/a.pdf": {Key: "kb/a.pdf", Body: []byte("PDFA"), ContentType: "application/pdf", ETag: "etag-a"}, - "kb/b.pdf": {Key: "kb/b.pdf", Body: []byte("PDFB"), ContentType: "application/pdf", ETag: "etag-b"}, - }, - } - svc.SetDatasourceUploadClientFactory(func(_ business.UploadDatasourceConfig, secret string) business.UploadContentClient { - gotSecret = secret + var gotKeyID, gotSecret string + var gotConfig objectstore.SourceConfig + fake := &fakeUploadBucket{contentType: "application/pdf", objects: []fakeObject{ + {key: "kb/a.pdf", etag: "etag-a", body: "PDFA"}, {key: "kb/b.pdf", etag: "etag-b", body: "PDFB"}, + }} + svc.SetDatasourceUploadBucketFactory(func(cfg objectstore.SourceConfig, keyID, secret string) objectstore.Bucket { + gotConfig, gotKeyID, gotSecret = cfg, keyID, secret return fake }) source := newUploadSource(t, svc) @@ -1544,12 +1620,16 @@ func TestRunDatasourceSync_UploadStreamsPerObject(t *testing.T) { if err != nil { t.Fatal(err) } - // Two object jobs, then the listing that closes the sync. - if enqueued != 2 || len(producer.jobs) != 3 || fake.listCalls != 1 || fake.fetchCalls != 2 { - t.Fatalf("enqueued=%d jobs=%d list=%d fetch=%d, want 2/3/1/2", enqueued, len(producer.jobs), fake.listCalls, fake.fetchCalls) + // Two object jobs, then the listing that closes the sync: one listing and + // one batched read. + if enqueued != 2 || len(producer.jobs) != 3 || fake.listCalls != 1 || fake.readCalls != 1 || len(fake.read) != 2 { + t.Fatalf("enqueued=%d jobs=%d list=%d read calls=%d objects read=%d, want 2/3/1/1/2", enqueued, len(producer.jobs), fake.listCalls, fake.readCalls, len(fake.read)) } - if gotSecret != "secretkey" { - t.Fatalf("secret handed to connector = %q, want the decrypted secret key", gotSecret) + if gotSecret != "secretkey" || gotKeyID != "AKIAEXAMPLE" { + t.Fatalf("credential bound to the store = %q/%q, want the source's key id and decrypted secret key", gotKeyID, gotSecret) + } + if gotConfig.Endpoint != "https://s3.us-east-1.amazonaws.com" || gotConfig.Bucket != "docs" || gotConfig.Prefix != "kb/" { + t.Fatalf("store bound to %+v, want the source's endpoint, bucket and prefix", gotConfig) } job := producer.jobs[0] if job.GetTopic() != "datasource.upload.sync" || !job.GetScope().GetGlobal() { @@ -1574,11 +1654,11 @@ func TestRunDatasourceSync_UploadStreamsPerObject(t *testing.T) { func TestRunDatasourceSync_UploadPropagatesFetchFailure(t *testing.T) { producer := &recordingProducer{} svc, _ := newDatasourceService(newDatasourceFakeStore(), producer, nil) - fake := &fakeUploadClient{ - entries: []objectstore.Entry{{Key: "kb/a.pdf", ETag: "e"}}, - fetchErr: map[string]error{"kb/a.pdf": errors.New("AccessDenied")}, + fake := &fakeUploadBucket{ + objects: []fakeObject{{key: "kb/a.pdf", etag: "e", body: "A"}}, + readErr: map[string]error{"kb/a.pdf": errors.New("AccessDenied")}, } - svc.SetDatasourceUploadClientFactory(func(business.UploadDatasourceConfig, string) business.UploadContentClient { return fake }) + useUploadBucket(svc, fake) source := newUploadSource(t, svc) takeConnectSync(t, producer, source) @@ -1590,20 +1670,16 @@ func TestRunDatasourceSync_UploadPropagatesFetchFailure(t *testing.T) { } } -// A listed object deleted before its fetch (404) is skipped best-effort while the -// rest still deliver; an oversized object is surfaced as an error. -func TestRunDatasourceSync_UploadSkipsVanishedAndSurfacesOversized(t *testing.T) { +// An object too large for the ingest inbox is known from its listed size: it is +// never read, the rest still deliver, and the sync surfaces it without a +// listing that would delete it. +func TestRunDatasourceSync_UploadSurfacesOversizedWithoutReadingIt(t *testing.T) { producer := &recordingProducer{} svc, _ := newDatasourceService(newDatasourceFakeStore(), producer, nil) - fake := &fakeUploadClient{ - entries: []objectstore.Entry{{Key: "ok.pdf", ETag: "e1"}, {Key: "gone.pdf", ETag: "e2"}, {Key: "huge.pdf", ETag: "e3"}}, - objects: map[string]objectstore.Object{"ok.pdf": {Key: "ok.pdf", Body: []byte("OK"), ETag: "e1"}}, - fetchErr: map[string]error{ - "gone.pdf": objectstore.ErrObjectNotFound, - "huge.pdf": objectstore.ErrObjectTooLarge, - }, - } - svc.SetDatasourceUploadClientFactory(func(business.UploadDatasourceConfig, string) business.UploadContentClient { return fake }) + fake := &fakeUploadBucket{objects: []fakeObject{ + {key: "kb/ok.pdf", etag: "e1", body: "OK"}, {key: "kb/huge.pdf", etag: "e3", size: 2 << 20}, + }} + useUploadBucket(svc, fake) source := newUploadSource(t, svc) takeConnectSync(t, producer, source) @@ -1611,10 +1687,11 @@ func TestRunDatasourceSync_UploadSkipsVanishedAndSurfacesOversized(t *testing.T) if err == nil { t.Fatal("the oversized object must surface as an error") } - // The vanished object is skipped and the deliverable one still enqueues; the - // undeliverable one keeps the sync from sending a listing that omits it. - if enqueued != 1 || len(producer.jobs) != 1 || producer.jobs[0].GetAttributes()["upload.key"] != "ok.pdf" { - t.Fatalf("enqueued=%d jobs=%d, want only ok.pdf delivered and no listing", enqueued, len(producer.jobs)) + if enqueued != 1 || len(producer.jobs) != 1 || producer.jobs[0].GetAttributes()["upload.key"] != "kb/ok.pdf" { + t.Fatalf("enqueued=%d jobs=%d, want only kb/ok.pdf delivered and no listing", enqueued, len(producer.jobs)) + } + if !reflect.DeepEqual(fake.read, []string{"kb/ok.pdf"}) { + t.Fatalf("read %v, want the oversized object never read", fake.read) } } @@ -1992,6 +2069,38 @@ func TestAddSourceNormalizesFileExtensions(t *testing.T) { } } +// TestAddSource_AdmitsAnObjectStorageSource is the admin path a composed +// consumer drives: object storage is on the envelope, so AddSource takes a +// source with a prefix and a key filter, seals its secret, enqueues its first +// sync, and that sync delivers only what the filter admits. +func TestAddSource_AdmitsAnObjectStorageSource(t *testing.T) { + producer := &recordingProducer{} + svc, _ := newDatasourceService(newDatasourceFakeStore(), producer, nil) + fake := &fakeUploadBucket{objects: []fakeObject{ + {key: "kb/guides/a.md", etag: "a", body: "A"}, {key: "kb/drafts/b.md", etag: "b", body: "B"}, + }} + useUploadBucket(svc, fake) + cfg := uploadConfig() + cfg.ExcludePatterns = []string{`^drafts/`} + source, err := svc.AddSource(context.Background(), "actor-1", business.AddSourceInput{ + OrgID: testOrg, Provider: business.DatasourceProviderUpload, CollectionLabel: "guides", Credential: "secretkey", Upload: cfg, + }) + if err != nil { + t.Fatalf("AddSource(upload) = %v, want admitted", err) + } + if source.Upload == nil || source.Upload.Prefix != "kb/" || !slices.Equal(source.Upload.ExcludePatterns, []string{`^drafts/`}) || + source.CredentialSecretRef == "" || source.CredentialSecretRef == "secretkey" { + t.Fatalf("stored source = %+v, want its config and a sealed secret", source) + } + takeConnectSync(t, producer, source) + if _, err := svc.RunDatasourceSync(context.Background(), source.ID); err != nil { + t.Fatal(err) + } + if got := uploadObjectJobs(producer); len(got) != 1 || got[0].GetAttributes()["upload.key"] != "kb/guides/a.md" { + t.Fatalf("delivered %v, want only kb/guides/a.md", got) + } +} + // TestAddSource_RefusesProvidersOffTheEnvelope holds the owner's settled rule: // a provider that does not pass the datasource conformance suite keeps its // existing sources running and takes no new one, refused before anything is @@ -2001,7 +2110,6 @@ func TestAddSource_RefusesProvidersOffTheEnvelope(t *testing.T) { svc, audit := newDatasourceService(store, &recordingProducer{}, nil) inputs := map[string]business.AddSourceInput{ business.DatasourceProviderCrawler: {OrgID: testOrg, Provider: business.DatasourceProviderCrawler, CollectionLabel: "guides", Crawler: crawlerConfig()}, - business.DatasourceProviderUpload: {OrgID: testOrg, Provider: business.DatasourceProviderUpload, CollectionLabel: "guides", Credential: "secretkey", Upload: uploadConfig()}, } for provider, in := range inputs { _, err := svc.AddSource(context.Background(), "actor-1", in) diff --git a/module/services/accounts/code/pkg/business/service.go b/module/services/accounts/code/pkg/business/service.go index cbfa992c7..cbf7602c7 100644 --- a/module/services/accounts/code/pkg/business/service.go +++ b/module/services/accounts/code/pkg/business/service.go @@ -6,6 +6,7 @@ import ( "accounts/pkg/auth" "accounts/pkg/datasource/apisource" "accounts/pkg/datasource/connector" + "accounts/pkg/datasource/objectstore" "accounts/pkg/email" "accounts/pkg/events" gen "accounts/pkg/gen/saas/accounts/v1" @@ -100,7 +101,7 @@ type Service struct { newGitHubClient func(token string) GitHubContentClient newAPIClient func(cfg APIDatasourceConfig, credential string) APIContentClient newCrawlerClient func(cfg CrawlerDatasourceConfig) CrawlerContentClient - newUploadClient func(cfg UploadDatasourceConfig, secretAccessKey string) UploadContentClient + newUploadBucket func(cfg objectstore.SourceConfig, accessKeyID, secretAccessKey string) objectstore.Bucket newOAuth2Refresh OAuth2RefreshFunc // refreshes an OAuth 2.0 API source's access token moduleProducer jobs.Producer // request-scoped, transactional outbox producer for the module-facing surface moduleJobStore jobs.Store // privileged worker store (claim/finalize) for the module-facing surface diff --git a/module/services/accounts/code/pkg/business/service_test.go b/module/services/accounts/code/pkg/business/service_test.go index 7b3af1c6e..fdc71c0e3 100644 --- a/module/services/accounts/code/pkg/business/service_test.go +++ b/module/services/accounts/code/pkg/business/service_test.go @@ -109,7 +109,7 @@ func runBusinessTests(m *testing.M) int { setupDone := testdb.MeasureSetup("business-db", []string{"store", "vault"}, 10*time.Minute) deps, err := sdk.WithDependencies(ctx, sdk.WithDebug(), - sdk.WithExcludedDependencies("cache"), + sdk.WithExcludedDependencies("object-storage", "cache"), // Keep this package-owned integration stack distinct from the outer // service runtime and the other package TestMain stacks. The SDK gives // each short-lived flow temporary host ports; the scope also isolates diff --git a/module/services/accounts/code/pkg/business/store.go b/module/services/accounts/code/pkg/business/store.go index 58545494d..f86c20b6d 100644 --- a/module/services/accounts/code/pkg/business/store.go +++ b/module/services/accounts/code/pkg/business/store.go @@ -4,6 +4,7 @@ import ( "context" "time" + "accounts/pkg/datasource/objectstore" gen "accounts/pkg/gen/saas/accounts/v1" ) @@ -128,6 +129,18 @@ type Store interface { // audit record names a source the same transaction actually deleted. DeleteDatasourceSource(ctx context.Context, orgID, id string) (*RemovedDatasourceSource, error) SetDatasourceSourceSynced(ctx context.Context, orgID, id string, syncedAt time.Time) error + // RecordDatasourceSnapshotSent records a sync that sent its closing + // snapshot: the sync time, and how many objects the snapshot named. + // DatasourceSourceLastSnapshot reads that count back; known is false for a + // source that never recorded one. Both MUST run inside the source's + // WithOrgTx. + RecordDatasourceSnapshotSent(ctx context.Context, orgID, id string, syncedAt time.Time, items int) error + DatasourceSourceLastSnapshot(ctx context.Context, orgID, id string) (items int, known bool, err error) + // PutDatasourceObjectManifest retains an object-storage source's listing + // under its version, and GetDatasourceObjectManifest reads one back (false + // once pruned). Both MUST run inside the source's WithOrgTx. + PutDatasourceObjectManifest(ctx context.Context, orgID, sourceID, version string, objects []objectstore.Item) error + GetDatasourceObjectManifest(ctx context.Context, orgID, sourceID, version string) ([]objectstore.Item, bool, error) // LockDatasourceSourceCredentialRef reads the source's current credential // envelope under a row lock (SELECT … FOR UPDATE) so a refresh-and-rotate // read-modify-write serializes against concurrent syncs of the same source. diff --git a/module/services/accounts/code/pkg/cataloggen/deployment_topology_test.go b/module/services/accounts/code/pkg/cataloggen/deployment_topology_test.go index f1b4d9dbe..6c204f272 100644 --- a/module/services/accounts/code/pkg/cataloggen/deployment_topology_test.go +++ b/module/services/accounts/code/pkg/cataloggen/deployment_topology_test.go @@ -71,16 +71,16 @@ func TestDeploymentTopologyIsDeterministicAndCurrent(t *testing.T) { require.Equal(t, string(readFixture(t, "testdata/network-policy.golden.yaml")), string(first.NetworkPolicy), "run: go generate ./pkg/cataloggen") require.Equal(t, string(readFixture(t, "testdata/mesh-policy.golden.yaml")), string(first.MeshPolicy), "run: go generate ./pkg/cataloggen") - require.Len(t, first.Catalog.GetServices(), 7) + require.Len(t, first.Catalog.GetServices(), 8) require.Len(t, first.Catalog.GetInterfaceEndpoints(), 7) - require.Len(t, first.Catalog.GetPublicEgress(), 3) + require.Len(t, first.Catalog.GetPublicEgress(), 4) endpointCount, dependencyCount := 0, 0 for _, service := range first.Catalog.GetServices() { endpointCount += len(service.GetEndpoints()) dependencyCount += len(service.GetDependencies()) } - require.Equal(t, 13, endpointCount) - require.Equal(t, 6, dependencyCount) + require.Equal(t, 14, endpointCount) + require.Equal(t, 7, dependencyCount) // Accounts REST stays private. Gateway REST is exported for module mints; // solutions resolve its separately exported workload listener. restVisibility := map[string]catalogv1.EndpointVisibility{ @@ -113,7 +113,7 @@ func TestDeploymentTopologyIsDeterministicAndCurrent(t *testing.T) { for _, grant := range first.Catalog.GetPublicEgress() { egress[grant.GetService()] = true } - require.Equal(t, map[string]bool{"accounts": true, "frontend": true, "marketing": true}, egress) + require.Equal(t, map[string]bool{"accounts": true, "frontend": true, "marketing": true, "object-storage": true}, egress) require.False(t, egress["auth-gateway"], "auth-gateway must have no public egress") accountsConnectExposed, accountsAuthorityExposed, gatewayRESTExposed := false, false, false for _, endpoint := range first.Catalog.GetInterfaceEndpoints() { @@ -134,7 +134,7 @@ func TestDeploymentTopologyIsDeterministicAndCurrent(t *testing.T) { require.True(t, accountsConnectExposed) require.True(t, accountsAuthorityExposed) require.True(t, gatewayRESTExposed) - require.Equal(t, 18, strings.Count(string(first.NetworkPolicy), "\nkind: NetworkPolicy\n")) + require.Equal(t, 20, strings.Count(string(first.NetworkPolicy), "\nkind: NetworkPolicy\n")) require.NotContains(t, string(first.NetworkPolicy), "allow-intra-namespace") for _, name := range []string{ "allow-accounts-from-dependents", "allow-auth-gateway-from-dependents", "allow-auth-gateway-to-dependencies", @@ -225,7 +225,7 @@ func TestGeneratedCodeflyAndNetworkManifestsParseStrictly(t *testing.T) { require.NoError(t, loadedModule.ValidateInterface(ctx)) loadedServices, err := loadedModule.LoadServices(ctx) require.NoError(t, err) - require.Len(t, loadedServices, 7) + require.Len(t, loadedServices, 8) moduleDocument := readFixture(t, "../../../../../module.codefly.yaml") var moduleEntry struct { @@ -237,7 +237,7 @@ func TestGeneratedCodeflyAndNetworkManifestsParseStrictly(t *testing.T) { require.NoError(t, yaml.Unmarshal(moduleDocument, &module)) _, err = module.Proto(ctx) require.NoError(t, err) - require.Len(t, module.ServiceReferences, 7) + require.Len(t, module.ServiceReferences, 8) for _, reference := range module.ServiceReferences { document := readFixture(t, filepath.Join("../../../../../services", reference.Name, "service.codefly.yaml")) @@ -279,7 +279,7 @@ func TestGeneratedCodeflyAndNetworkManifestsParseStrictly(t *testing.T) { require.False(t, names[document.Metadata.Name], "duplicate NetworkPolicy %s", document.Metadata.Name) names[document.Metadata.Name] = true } - require.Len(t, names, 18) + require.Len(t, names, 20) require.True(t, names["allow-istio-ingress-to-marketing"]) require.True(t, names["allow-istio-ingress-to-frontend"]) require.False(t, names["allow-istio-ingress-to-auth-gateway"]) diff --git a/module/services/accounts/code/pkg/cataloggen/testdata/network-policy.golden.yaml b/module/services/accounts/code/pkg/cataloggen/testdata/network-policy.golden.yaml index 1e7e3d0bd..5afe881df 100644 --- a/module/services/accounts/code/pkg/cataloggen/testdata/network-policy.golden.yaml +++ b/module/services/accounts/code/pkg/cataloggen/testdata/network-policy.golden.yaml @@ -174,6 +174,26 @@ spec: --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy +metadata: + name: allow-object-storage-from-dependents + namespace: saas-starter +spec: + podSelector: + matchLabels: + app: object-storage + policyTypes: + - Ingress + ingress: + - from: + - podSelector: + matchLabels: + app: accounts + ports: + - protocol: TCP + port: 9090 +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy metadata: name: allow-store-from-dependents namespace: saas-starter @@ -231,6 +251,13 @@ spec: ports: - protocol: TCP port: 6379 + - to: + - podSelector: + matchLabels: + app: object-storage + ports: + - protocol: TCP + port: 9090 - to: - podSelector: matchLabels: @@ -490,3 +517,55 @@ spec: ports: - protocol: TCP port: 443 +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-object-storage-public-egress + namespace: saas-starter +spec: + podSelector: + matchLabels: + app: object-storage + policyTypes: + - Egress + egress: + - to: + - ipBlock: + cidr: 0.0.0.0/0 + except: + - 0.0.0.0/8 + - 10.0.0.0/8 + - 100.64.0.0/10 + - 127.0.0.0/8 + - 169.254.0.0/16 + - 172.16.0.0/12 + - 192.0.0.0/24 + - 192.0.2.0/24 + - 192.31.196.0/24 + - 192.52.193.0/24 + - 192.88.99.0/24 + - 192.168.0.0/16 + - 192.175.48.0/24 + - 198.18.0.0/15 + - 198.51.100.0/24 + - 203.0.113.0/24 + - 224.0.0.0/4 + - 240.0.0.0/4 + - ipBlock: + cidr: ::/0 + except: + - ::/128 + - ::1/128 + - 64:ff9b::/96 + - 64:ff9b:1::/48 + - 100::/64 + - 2001::/23 + - 2001:db8::/32 + - 2002::/16 + - fc00::/7 + - fe80::/10 + - ff00::/8 + ports: + - protocol: TCP + port: 443 diff --git a/module/services/accounts/code/pkg/datasource/connector/CONFORMANCE.md b/module/services/accounts/code/pkg/datasource/connector/CONFORMANCE.md index f175f11a8..d733b87ed 100644 --- a/module/services/accounts/code/pkg/datasource/connector/CONFORMANCE.md +++ b/module/services/accounts/code/pkg/datasource/connector/CONFORMANCE.md @@ -2,14 +2,15 @@ The datasource connector model ships with an envelope (`envelope.go`), a descriptor-driven registry (`registry.go`) and a conformance suite -(`connectortest`). **GitHub is the only connector that passes it.** The generic -API, the crawler and object storage run on engines of their own, are registered +(`connectortest`). **Two connectors pass it: GitHub and object storage.** The +generic API and the crawler run on engines of their own, are registered non-conformant with a gap, keep their existing sources running, and take no new ones. -A suite with one conformant implementation has not yet shown the model -generalises, which was the reason for having one. This document is the honest -version of that: per connector, what it already honours, what remains, and what +A suite with one conformant implementation had not shown the model generalises, +which was the reason for having one; object storage is the second, on a store +that answers "what is there now" rather than "what changed since". This document +is the honest record: per connector, what it honours, what remains, and what closing it would actually take. It is held to `pkg/business/datasource_connectors.go` by `connector_conformance_record_test.go`, so a provider cannot be registered here without a section, or given a section that names no registered provider. @@ -39,28 +40,76 @@ resync on rewritten history, deletions, provenance, bulk fetch in a bounded number of provider calls, typed rate limits with a reset time, typed refusals for an over-limit item or batch, and cancellation. -## upload — object storage +## upload — object storage, conformant -**Registered gap:** no cursor, so every sync re-sends every object, and one -fetch per object; deletions ARE applied, through the complete listing that -closes a whole-folder sync. +**Conformant.** `objectstore.FilesConnector` passes `connectortest.RunFiles` +through `objectstore.ServiceBucket` — the host's client of the composed +object-storage service (codefly-dev/service-object-storage, its per-call bucket +binding and batched read from #108) — over an in-process fake of the service's +gRPC API, so every clause is measured on the calls the host makes +(`objectstore/connector_conformance_test.go`). The host signs, dials and guards +nothing for a bucket; `client.go`, its own store client, is deleted. It is +registered through `connector.Budgeted`, metered per provider credential like +GitHub, and `AdmitNewSource` admits it; the host admits a new source only where +a composed object-storage service can read a bucket (below). -| Clause | Where it stands | +| Clause | How it is honoured | | --- | --- | -| 1 identity and tenancy | Honoured in substance. The object key is the provider's stable id and the ETag is its version; every enqueued job names the source, org and boundary. Not expressed through `connector.Key`/`Item`, because the connector is not on the envelope. | -| 2 readers | No per-object access list is read, so every object is readable by whoever may read the boundary — which is `ReadersSourceScoped`, the model's own answer for a provider like this. Nothing to close. | -| 4 versions and change sets | **The remaining gap.** There is no version to diff from, so `runUploadSync` re-lists and re-fetches the whole prefix on every sync. Deletions are not missing: a sync that saw the whole folder and delivered every object in it closes with the complete listing of what it delivered, and the store tombstones the rest; a failed or truncated listing sends none, so a partial view can never delete. | -| 5 provenance | Key and ETag ride on each delivery; not in the envelope's shape. | -| 6 budget and backpressure | No declared `Budget`, and the connector is not metered by `datasource_credential_budgets`, so an object-storage sync cannot yield to a person's sync the way a GitHub one does. | -| 8 bulk only | One `GetObject` per object. The listing is one call; the fetches are not. | - -**What closing it takes.** A `FilesConnector` over S3 whose version is a digest -of the listing (key → ETag) and whose `Changes(from)` diffs the current listing -against the one `from` names — which needs the previous listing to be retained, -since S3 has no "what changed since" API. That is a store-shaped change, not -just a connector: the host would have to keep the last manifest per source. Not -done here because it is a design question about where that manifest lives, not a -gap in a flow that already exists. +| 1 identity and tenancy | `connector.KeyFor(source, object key)`: the object key is the item id and its ETag its version. | +| 2 readers | `ReadersSourceScoped`: a bucket has no per-object access list, so every object is readable by whoever may read the source's collection. | +| 4 versions and change sets | A version is a digest of the listing within the source's key filter and object cap, under its binding; the host retains the listing each version names in `datasource_object_manifests` (migration 59; the newest four per source), and `Changes(from)` diffs against it. The host's own sync still takes a complete snapshot each time and re-sends every object, closing with the listing that tombstones the rest; delivering only what changed means moving `runUploadSync` onto the change-set path, which changes what the consuming module receives, so it is its own change. | +| 5 provenance | Source, org, boundary, version, key and ETag on every fetched file. | +| 6 budget and backpressure | A declared `Budget`, metered by `datasource_credential_budgets` through `connector.Budgeted`, keyed by the credential — a digest of the store's origin and the access key id, derived from the source's configuration on every use and never stored — so every source reading with one key at one store spends one quota and honours one provider throttle, whenever it was connected. A sync a person asks for may spend the reserved share; the scheduled pass yields. | +| 8 bulk only | A batch is one `GetMany` per `max_get_many_items` (1000, the connector's batch): 120 objects cost one call after the listing. | + +**How it reads a store.** + +- **The binding.** `objectstore.ServiceBucket` names the source's endpoint, + region and bucket and carries its access key for each call; the service + never stores it. The binding carries **no prefix**: the service normalizes a + binding's prefix as a folder (no leading or trailing `/`), while a source's + prefix is a literal S3 key prefix — `/reports/` is not `reports/`, and `/` is + not the whole bucket. Every listing filters on the literal prefix, which the + service passes to the store unchanged, and every key travels whole, which is + what the consuming module has always been sent. A listed key outside the + literal prefix, or a page past the service's own page size, refuses the whole + listing. Before it lists or reads, it asks `Capabilities` under the binding + and refuses a service that reports no bound caps or no batch size — an older + one would serve its own bucket as the tenant's. +- **Every call has a deadline**: 30 seconds per listing page or capability + question, five minutes per batched read. A store that answers and then + stalls does not hold a sync, and its worker's lease, past it. +- **Admission needs a service that can read a bucket.** At start the host asks + the composed service, unbound, whether it reads a batch; one that does not — + the released 0.0.18 agent, or none composed — leaves object storage + unavailable: the catalog reports `accepts_new_sources: false` and `AddSource` + refuses by name. +- **The guard and the caps are the service's.** A non-public endpoint is + refused there by name (`PERMISSION_DENIED`, reported as the envelope's + credential refusal: it is the source's configuration to fix either way). A + bound read is capped at 5 MiB, the connector's `MaxObjectBytes`, so an object + over it is refused from its listed size before it is read; one the service + still answers `TOO_LARGE` is the envelope's `ErrItemTooLarge`. A listing is + paged at the service's page size and bounded at 20 pages; past that it is + `ErrListingIncomplete`, never a partial version. **Completeness is the + service's to report:** the host takes an empty `next_page_token` as the end + of the listing, so a store that answers "truncated" with no continuation + token must be refused by the service, which owns the S3 parsing — the host + cannot recover a truncation signal the service discards. +- **The scope is the connector's.** The key filter and the object cap are the + source's `objectstore.Scope`, applied to the listing before its digest: an + excluded object is never read and never counts against the cap, and a + listing the cap cuts is `ErrListingIncomplete`, which fails the sync + terminally with nothing delivered. +- **A sync never empties a source that has files.** A host-owned invariant, + whatever any service answers: a listing that comes back empty for a source + whose last closing snapshot named objects is refused + (`ErrUploadEmptyListingRefused`) and closes nothing. The per-key scope check + cannot catch an empty answer for the wrong scope; this rule does. +- **A fetch is at a version.** Each object of a batch is read at `If-Match` its + listed ETag; one rewritten or deleted since (`CHANGED`, `NOT_FOUND`) is + `ErrItemNotFound`, which fails the sync without a listing, and the next sync + lists afresh. ## crawler — documentation website @@ -121,8 +170,9 @@ required path parameters, escaping, query/body routing, and GET body refusal. limits, token endpoint guards, and transport disclosure. `apisource/guard_test.go` holds the egress guard to the shared `auth/egresstest` vectors, one named case per non-global range, and proves the operation dial and every token exchange -run the guard on the address being dialed; `crawler/guard_test.go` and -`objectstore/guard_test.go` do the same for those connectors. +run the guard on the address being dialed; `crawler/guard_test.go` does the +same for the crawler. Object storage dials nothing: its guard is the +object-storage service's. `connectortest.RunOperations`, run by `business/TestAPIOperationsConformance`, checks the host call path for unknown operations, schema and path refusals, GET body refusal, output caps, typed 429 responses and a lost mutation reply @@ -141,11 +191,10 @@ not qualify database commit/rollback or cross-process serialization. | Connector | Conformant | Blocked by the interface | Deletions | Cursor | Budget | | --- | --- | --- | --- | --- | --- | | github | yes | — | yes | yes | declared and metered | -| upload | no | no (files) | **yes** | no | no | +| upload | yes | — | yes | yes (the host's sync still re-sends every object) | declared and metered | | crawler | no | yes (pages) | no | no | no | | api | no | yes (records) | no | no | no | -Object storage is the nearest: it is already on an admissible interface and -already applies deletions, and one cursor stands between it and a second -conformant implementation. That is what would demonstrate the model generalises, -and it is the one worth doing next. +Object storage is the second conformant implementation, which is what shows the +model generalises: the same envelope over a store with no history of its own, +where the cursor is a listing the host retains. diff --git a/module/services/accounts/code/pkg/datasource/connector/budget.go b/module/services/accounts/code/pkg/datasource/connector/budget.go index ddd91e6e7..715ef3d63 100644 --- a/module/services/accounts/code/pkg/datasource/connector/budget.go +++ b/module/services/accounts/code/pkg/datasource/connector/budget.go @@ -66,6 +66,10 @@ type budgeted struct { func (b *budgeted) Descriptor() Descriptor { return b.inner.Descriptor() } +// Unwrap is the connector being metered, so a check of what implements a +// registered connector reaches the implementation rather than the meter. +func (b *budgeted) Unwrap() FilesConnector { return b.inner } + func (b *budgeted) acquire(ctx context.Context, src Source) error { return b.scheduler.Acquire(ctx, src, b.inner.Descriptor().Budget, PriorityFrom(ctx)) } diff --git a/module/services/accounts/code/pkg/datasource/objectstore/client.go b/module/services/accounts/code/pkg/datasource/objectstore/client.go deleted file mode 100644 index 71655d92c..000000000 --- a/module/services/accounts/code/pkg/datasource/objectstore/client.go +++ /dev/null @@ -1,390 +0,0 @@ -// Package objectstore is the S3-compatible object-storage datasource connector: -// given an endpoint, region, bucket, prefix, and AWS-style credentials, it lists -// the objects under the prefix (S3 ListObjectsV2) and fetches each one -// (GetObject), signing every request with AWS Signature Version 4. It addresses -// objects path-style ({endpoint}/{bucket}/{key}) so it works with MinIO and -// other S3-compatible stores as well as AWS S3. It holds no persistence or -// key-management concerns of its own — the business layer decrypts the secret -// access key and hands it in plaintext per client. -// -// The endpoint is tenant-supplied, so every dial is guarded against SSRF: a URL -// that resolves to any address that is not on the public internet is refused, -// on the initial request and on any redirect, checked against the concrete IP -// being connected to (not the hostname) so DNS rebinding cannot slip past the -// check. "Not on the public internet" is auth.IsPublicDestination, read from -// the IANA special-purpose registries; a declared host a meshed cell resolves -// into its mesh egress range is admitted by the same shared rule -// (auth/mesh_egress.go). -package objectstore - -import ( - "context" - "crypto/hmac" - "crypto/sha256" - "encoding/hex" - "encoding/xml" - "errors" - "fmt" - "io" - "net" - "net/http" - "net/url" - "strings" - "syscall" - "time" - - "accounts/pkg/auth" -) - -// maxResponseBytes bounds one listing or object body so a pathological endpoint -// cannot exhaust memory or overflow the downstream job payload limit. -const maxResponseBytes = 5 * 1024 * 1024 - -// defaultMaxObjects caps how many objects one Fetch returns when the source does -// not configure a limit; maxObjectsCeiling is the hard upper bound regardless of -// configuration. -const ( - defaultMaxObjects = 100 - maxObjectsCeiling = 1000 -) - -const maxRedirects = 3 - -// emptyPayloadHash is the SHA-256 of the empty payload, used as -// x-amz-content-sha256 for the GET requests this connector issues (they carry no -// body). -const emptyPayloadHash = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" - -// ErrBlockedAddress is returned when a request target (or a redirect) resolves to -// a non-public IP address. It is deliberately opaque so a tenant cannot use the -// connector to map the cluster's internal network. -var ErrBlockedAddress = errors.New("objectstore: request target is not a public address") - -// ErrObjectNotFound reports that a listed key returned 404 on GET — it was -// deleted between the listing and the fetch. The caller may skip it best-effort. -var ErrObjectNotFound = errors.New("objectstore: object not found") - -// ErrObjectTooLarge reports that an object body exceeds the connector's transport -// ceiling. It is distinct from a generic failure so the caller can surface an -// undeliverable object rather than silently discard it. -var ErrObjectTooLarge = errors.New("objectstore: object exceeds the maximum size") - -// Config is the non-secret configuration of one object-storage source. -type Config struct { - Endpoint string // absolute http(s) base URL, e.g. https://s3.us-east-1.amazonaws.com or a MinIO endpoint - Region string // e.g. "us-east-1" - Bucket string - Prefix string // key prefix to list under; may be empty - AccessKeyID string - MaxObjects int // 0 means the default cap -} - -// Object is one fetched object: its key, body bytes, content type, and ETag. -type Object struct { - Key string - Body []byte - ContentType string - ETag string -} - -// Client lists and fetches the objects of a single object-storage source. -type Client struct { - cfg Config - secret string - base *url.URL - baseErr error - http *http.Client -} - -// dialGuard builds the Control hook of every dial this package makes, from -// the "host:port" the client asked for. It is always guardDial in production; -// it is a variable only so a test can reach an httptest listener on loopback, -// or wrap guardDial in a recorder. -var dialGuard = guardDial - -// dialResolver resolves the requested host before the dial. It is -// net.DefaultResolver in production; a variable only so a test can answer with -// the address it needs. -var dialResolver auth.Resolver = net.DefaultResolver - -// guardDial refuses to open a connection the egress rule does not admit. Its -// hook runs with the concrete IP:port being dialed, so it blocks the exact -// address the connection would reach — closing the DNS-rebinding gap a -// hostname-only check leaves open. The judgement is -// auth.AdmitsDestinationDialAddress, the rule the host's guarded egress paths -// share: a public address, or a declared host's address in a meshed cell's mesh -// egress range. An address it cannot parse is refused. -func guardDial(requested string) auth.DialControl { - return func(_, address string, _ syscall.RawConn) error { - if !auth.AdmitsDestinationDialAddress(requested, address) { - return ErrBlockedAddress - } - return nil - } -} - -// New returns a Client for cfg, authenticating with the plaintext secret access -// key. The HTTP client blocks connections to non-public addresses. -func New(cfg Config, secretAccessKey string) *Client { - transport := &http.Transport{ - DialContext: auth.GuardedDialContext(net.Dialer{Timeout: 10 * time.Second}, dialResolver, - // Indirect through the dialGuard var so a test override is honored. - func(requested string) auth.DialControl { return dialGuard(requested) }), - } - c := &Client{ - cfg: cfg, - secret: secretAccessKey, - http: &http.Client{ - Timeout: 30 * time.Second, - Transport: transport, - CheckRedirect: func(_ *http.Request, via []*http.Request) error { - if len(via) >= maxRedirects { - return errors.New("objectstore: too many redirects") - } - return nil - }, - }, - } - base, err := url.Parse(strings.TrimSpace(cfg.Endpoint)) - switch { - case err != nil: - c.baseErr = fmt.Errorf("objectstore: invalid endpoint: %w", err) - case base.Scheme != "http" && base.Scheme != "https": - c.baseErr = errors.New("objectstore: endpoint must be http or https") - case base.Host == "": - c.baseErr = errors.New("objectstore: endpoint must have a host") - default: - c.base = base - } - return c -} - -// Entry is one row of a ListObjectsV2 result: an object key and its ETag. -type Entry struct { - Key string - ETag string -} - -// Listing is one List result. Complete is true only when the entries are every -// object under the prefix: the store answered with IsTruncated false and the -// object cap cut nothing. A caller that treats absence from a listing as a -// deletion must act only on a complete one. -type Listing struct { - Entries []Entry - Complete bool -} - -// List issues one ListObjectsV2 request and returns its entries, bounded by the -// configured object cap. Listing failure (including a non-2xx list) is a returned -// error so a credential or permission problem surfaces rather than reading as an -// empty bucket. -func (c *Client) List(ctx context.Context) (Listing, error) { - if c.baseErr != nil { - return Listing{}, c.baseErr - } - u := *c.base - u.Path = "/" + c.cfg.Bucket - u.RawPath = "/" + rfc3986Escape(c.cfg.Bucket) - // Canonical query string: sorted key=value, RFC3986-encoded. "list-type" - // sorts before "prefix", so this is already canonical order. - u.RawQuery = "list-type=2&prefix=" + rfc3986Escape(c.cfg.Prefix) - - body, status, _, err := c.do(ctx, &u) - if err != nil { - return Listing{}, err - } - if status < 200 || status >= 300 { - return Listing{}, fmt.Errorf("objectstore: list returned status %d", status) - } - var parsed listBucketResult - if err := xml.Unmarshal(body, &parsed); err != nil { - return Listing{}, fmt.Errorf("objectstore: parse listing: %w", err) - } - limit := c.cfg.MaxObjects - if limit <= 0 { - limit = defaultMaxObjects - } - if limit > maxObjectsCeiling { - limit = maxObjectsCeiling - } - entries := make([]Entry, 0, len(parsed.Contents)) - for _, e := range parsed.Contents { - if len(entries) >= limit { - break - } - entries = append(entries, Entry{Key: e.Key, ETag: e.ETag}) - } - return Listing{Entries: entries, Complete: !parsed.IsTruncated && len(parsed.Contents) <= limit}, nil -} - -// listEntry is one row of a ListBucketResult. -type listEntry struct { - Key string `xml:"Key"` - ETag string `xml:"ETag"` -} - -type listBucketResult struct { - XMLName xml.Name `xml:"ListBucketResult"` - IsTruncated bool `xml:"IsTruncated"` - Contents []listEntry `xml:"Contents"` -} - -// Fetch retrieves one object by key. The caller drives it per Entry from List, so -// only one object body is ever held in memory at a time. A 404 returns -// ErrObjectNotFound (the key was deleted between listing and fetch); a body over -// the transport ceiling returns ErrObjectTooLarge; any other non-2xx or transport -// error is returned so the caller can treat it as a real failure. -func (c *Client) Fetch(ctx context.Context, key string) (Object, error) { - if c.baseErr != nil { - return Object{}, c.baseErr - } - u := *c.base - u.Path = "/" + c.cfg.Bucket + "/" + key - // Path-escape each key segment but keep the "/" separators. - u.RawPath = "/" + rfc3986Escape(c.cfg.Bucket) + "/" + encodeKeyPath(key) - u.RawQuery = "" - - body, status, header, err := c.do(ctx, &u) - if err != nil { - if errors.Is(err, errBodyTooLarge) { - return Object{}, ErrObjectTooLarge - } - return Object{}, err - } - if status == http.StatusNotFound { - return Object{}, ErrObjectNotFound - } - if status < 200 || status >= 300 { - return Object{}, fmt.Errorf("objectstore: get %q returned status %d", key, status) - } - return Object{ - Key: key, - Body: body, - ContentType: header.Get("Content-Type"), - ETag: header.Get("ETag"), - }, nil -} - -// errBodyTooLarge is the internal signal do() raises when a response body exceeds -// the transport ceiling; Fetch translates it to the exported ErrObjectTooLarge. -var errBodyTooLarge = errors.New("objectstore: response exceeds the maximum size") - -// do builds, signs, and sends a GET request for u, returning the bounded body, -// the status code, and the response header. An over-large body is errBodyTooLarge. -func (c *Client) do(ctx context.Context, u *url.URL) ([]byte, int, http.Header, error) { - req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.base.String(), nil) - if err != nil { - return nil, 0, nil, fmt.Errorf("objectstore: build request: %w", err) - } - // Use the exact escaped path/query we built, not a re-parsed copy. - req.URL = u - req.Host = u.Host - c.sign(req, emptyPayloadHash, time.Now()) - - resp, err := c.http.Do(req) - if err != nil { - return nil, 0, nil, fmt.Errorf("objectstore: request: %w", err) - } - defer resp.Body.Close() - - body, err := io.ReadAll(io.LimitReader(resp.Body, maxResponseBytes+1)) - if err != nil { - return nil, 0, nil, fmt.Errorf("objectstore: read response: %w", err) - } - if len(body) > maxResponseBytes { - return nil, 0, nil, errBodyTooLarge - } - return body, resp.StatusCode, resp.Header, nil -} - -// sign applies AWS Signature Version 4 (service "s3") to req in place. It signs -// the host, x-amz-content-sha256, and x-amz-date headers. -func (c *Client) sign(req *http.Request, payloadHash string, now time.Time) { - amzDate := now.UTC().Format("20060102T150405Z") - dateStamp := now.UTC().Format("20060102") - - host := req.Host - if host == "" { - host = req.URL.Host - } - req.Header.Set("x-amz-date", amzDate) - req.Header.Set("x-amz-content-sha256", payloadHash) - - canonicalURI := req.URL.EscapedPath() - if canonicalURI == "" { - canonicalURI = "/" - } - canonicalHeaders := "host:" + host + "\n" + - "x-amz-content-sha256:" + payloadHash + "\n" + - "x-amz-date:" + amzDate + "\n" - signedHeaders := "host;x-amz-content-sha256;x-amz-date" - - canonicalRequest := strings.Join([]string{ - req.Method, - canonicalURI, - req.URL.RawQuery, // already canonical: sorted, RFC3986-encoded - canonicalHeaders, - signedHeaders, - payloadHash, - }, "\n") - - scope := dateStamp + "/" + c.cfg.Region + "/s3/aws4_request" - stringToSign := strings.Join([]string{ - "AWS4-HMAC-SHA256", - amzDate, - scope, - sha256Hex([]byte(canonicalRequest)), - }, "\n") - - key := signingKey(c.secret, dateStamp, c.cfg.Region, "s3") - signature := hex.EncodeToString(hmacSHA256(key, []byte(stringToSign))) - - req.Header.Set("Authorization", "AWS4-HMAC-SHA256 Credential="+c.cfg.AccessKeyID+"/"+scope+ - ", SignedHeaders="+signedHeaders+", Signature="+signature) -} - -// signingKey derives the SigV4 signing key: AWS4+secret, then successively keyed -// by the date stamp, region, service, and the literal "aws4_request". -func signingKey(secret, dateStamp, region, service string) []byte { - kDate := hmacSHA256([]byte("AWS4"+secret), []byte(dateStamp)) - kRegion := hmacSHA256(kDate, []byte(region)) - kService := hmacSHA256(kRegion, []byte(service)) - return hmacSHA256(kService, []byte("aws4_request")) -} - -func hmacSHA256(key, data []byte) []byte { - h := hmac.New(sha256.New, key) - h.Write(data) - return h.Sum(nil) -} - -func sha256Hex(b []byte) string { - sum := sha256.Sum256(b) - return hex.EncodeToString(sum[:]) -} - -// encodeKeyPath path-escapes each "/"-separated segment of an S3 key while -// keeping the separators, per the S3 canonical-URI rules. -func encodeKeyPath(key string) string { - segments := strings.Split(key, "/") - for i, s := range segments { - segments[i] = rfc3986Escape(s) - } - return strings.Join(segments, "/") -} - -// rfc3986Escape percent-encodes s per RFC 3986, leaving only the unreserved set -// (A-Z a-z 0-9 - _ . ~) unescaped — the encoding SigV4 canonicalization expects. -func rfc3986Escape(s string) string { - var b strings.Builder - for i := 0; i < len(s); i++ { - ch := s[i] - if (ch >= 'A' && ch <= 'Z') || (ch >= 'a' && ch <= 'z') || - (ch >= '0' && ch <= '9') || ch == '-' || ch == '_' || ch == '.' || ch == '~' { - b.WriteByte(ch) - continue - } - b.WriteString(fmt.Sprintf("%%%02X", ch)) - } - return b.String() -} diff --git a/module/services/accounts/code/pkg/datasource/objectstore/client_test.go b/module/services/accounts/code/pkg/datasource/objectstore/client_test.go deleted file mode 100644 index 65dbc0a17..000000000 --- a/module/services/accounts/code/pkg/datasource/objectstore/client_test.go +++ /dev/null @@ -1,344 +0,0 @@ -package objectstore - -import ( - "context" - "encoding/hex" - "errors" - "net/http" - "net/http/httptest" - "regexp" - "strings" - "syscall" - "testing" - - "accounts/pkg/auth" -) - -// allowDial overrides the SSRF dial guard so tests can reach httptest's -// 127.0.0.1 listener, restoring the real guard when the test finishes. -func allowDial(t *testing.T) { - t.Helper() - saved := dialGuard - dialGuard = func(string) auth.DialControl { return func(_, _ string, _ syscall.RawConn) error { return nil } } - t.Cleanup(func() { dialGuard = saved }) -} - -func TestFetchHappyPath(t *testing.T) { - allowDial(t) - - const bucket = "my-bucket" - const prefix = "docs/" - var sawAuth, sawDate bool - - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if r.Header.Get("Authorization") != "" { - sawAuth = true - } - if r.Header.Get("x-amz-date") != "" { - sawDate = true - } - if a := r.Header.Get("Authorization"); a != "" && !strings.HasPrefix(a, "AWS4-HMAC-SHA256") { - t.Errorf("Authorization not AWS4-HMAC-SHA256: %q", a) - } - if r.URL.Query().Get("list-type") == "2" { - w.Header().Set("Content-Type", "application/xml") - w.Write([]byte(` - - my-bucket - docs/ - docs/a.txt"etag-a" - docs/b.txt"etag-b" -`)) - return - } - switch r.URL.Path { - case "/my-bucket/docs/a.txt": - w.Header().Set("Content-Type", "text/plain") - w.Header().Set("ETag", `"etag-a"`) - w.Write([]byte("alpha")) - case "/my-bucket/docs/b.txt": - w.Header().Set("Content-Type", "text/markdown") - w.Header().Set("ETag", `"etag-b"`) - w.Write([]byte("bravo")) - default: - http.Error(w, "not found", http.StatusNotFound) - } - })) - defer srv.Close() - - c := New(Config{ - Endpoint: srv.URL, - Region: "us-east-1", - Bucket: bucket, - Prefix: prefix, - AccessKeyID: "AKIAIOSFODNN7EXAMPLE", - }, "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY") - - listing, err := c.List(context.Background()) - if err != nil { - t.Fatalf("List: %v", err) - } - entries := listing.Entries - if !listing.Complete { - t.Fatal("a listing the store did not truncate reported itself incomplete") - } - if len(entries) != 2 { - t.Fatalf("want 2 entries, got %d", len(entries)) - } - byKey := map[string]Object{} - for _, e := range entries { - o, err := c.Fetch(context.Background(), e.Key) - if err != nil { - t.Fatalf("Fetch %s: %v", e.Key, err) - } - byKey[o.Key] = o - } - a, ok := byKey["docs/a.txt"] - if !ok { - t.Fatalf("missing docs/a.txt in %v", byKey) - } - if string(a.Body) != "alpha" || a.ContentType != "text/plain" || a.ETag != `"etag-a"` { - t.Errorf("a mismatch: body=%q ct=%q etag=%q", a.Body, a.ContentType, a.ETag) - } - b, ok := byKey["docs/b.txt"] - if !ok { - t.Fatalf("missing docs/b.txt in %v", byKey) - } - if string(b.Body) != "bravo" || b.ContentType != "text/markdown" || b.ETag != `"etag-b"` { - t.Errorf("b mismatch: body=%q ct=%q etag=%q", b.Body, b.ContentType, b.ETag) - } - if !sawAuth { - t.Error("server never saw an Authorization header") - } - if !sawDate { - t.Error("server never saw an x-amz-date header") - } -} - -// TestAuthorizationHeaderStructure asserts the shape of the Authorization header -// a real signed request carries. -func TestAuthorizationHeaderStructure(t *testing.T) { - allowDial(t) - - var auth string - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - auth = r.Header.Get("Authorization") - w.Header().Set("Content-Type", "application/xml") - w.Write([]byte(``)) - })) - defer srv.Close() - - c := New(Config{ - Endpoint: srv.URL, - Region: "us-east-1", - Bucket: "b", - AccessKeyID: "AKIAIOSFODNN7EXAMPLE", - }, "secret") - if _, err := c.List(context.Background()); err != nil { - t.Fatalf("List: %v", err) - } - - re := regexp.MustCompile(`^AWS4-HMAC-SHA256 Credential=AKIA[^ ]+/\d{8}/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=[0-9a-f]{64}$`) - if !re.MatchString(auth) { - t.Fatalf("Authorization header does not match expected structure: %q", auth) - } -} - -// TestSigV4KnownAnswer verifies the signing-key derivation and final signature -// against AWS's published S3 "GET Object" SigV4 example (service s3, region -// us-east-1, secret wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY). -// https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html -// -// The canonical request below is the documented one; its SHA-256 is AWS's -// published intermediate hash 7344ae5b...946972, and the resulting signature is -// the value AWS's own worked example computes. -func TestSigV4KnownAnswer(t *testing.T) { - canonicalRequest := strings.Join([]string{ - "GET", - "/test.txt", - "", - "host:examplebucket.s3.amazonaws.com", - "range:bytes=0-9", - "x-amz-content-sha256:" + emptyPayloadHash, - "x-amz-date:20130524T000000Z", - "", - "host;range;x-amz-content-sha256;x-amz-date", - emptyPayloadHash, - }, "\n") - - crHash := sha256Hex([]byte(canonicalRequest)) - const wantCRHash = "7344ae5b7ee6c3e7e6b0fe0640412a37625d1fbfff95c48bbb2dc43964946972" - if crHash != wantCRHash { - t.Fatalf("canonical-request hash mismatch:\n got %s\nwant %s", crHash, wantCRHash) - } - - stringToSign := strings.Join([]string{ - "AWS4-HMAC-SHA256", - "20130524T000000Z", - "20130524/us-east-1/s3/aws4_request", - crHash, - }, "\n") - - key := signingKey("wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY", "20130524", "us-east-1", "s3") - got := hex.EncodeToString(hmacSHA256(key, []byte(stringToSign))) - - const want = "67fe34c8530db585abddc51067328adfedb6e42487d2566dc7d927d6e2722900" - if got != want { - t.Fatalf("signature mismatch:\n got %s\nwant %s", got, want) - } -} - -// TestSigV4TestSuiteGetVanilla anchors the signing routine to the independent -// AWS SigV4 test-suite "get-vanilla" vector (service "service"), whose published -// signature is a fixed known answer — proving signingKey/hmac/sha256 are correct -// independently of this connector's own request-building. -func TestSigV4TestSuiteGetVanilla(t *testing.T) { - canonicalRequest := strings.Join([]string{ - "GET", - "/", - "", - "host:example.amazonaws.com", - "x-amz-date:20150830T123600Z", - "", - "host;x-amz-date", - emptyPayloadHash, - }, "\n") - stringToSign := strings.Join([]string{ - "AWS4-HMAC-SHA256", - "20150830T123600Z", - "20150830/us-east-1/service/aws4_request", - sha256Hex([]byte(canonicalRequest)), - }, "\n") - key := signingKey("wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY", "20150830", "us-east-1", "service") - got := hex.EncodeToString(hmacSHA256(key, []byte(stringToSign))) - - const want = "5fa00fa31553b73ebf1942676e86291e8372ff2a2260956d9b8aae1d763fbf31" - if got != want { - t.Fatalf("signature mismatch:\n got %s\nwant %s", got, want) - } -} - -// TestEmptyPayloadHash guards the well-known empty-string SHA-256 constant used -// for x-amz-content-sha256 on the bodyless GET requests. -func TestEmptyPayloadHash(t *testing.T) { - if got := sha256Hex(nil); got != emptyPayloadHash { - t.Fatalf("empty payload hash mismatch: got %s want %s", got, emptyPayloadHash) - } -} - -// TestFetchDistinguishesFailures proves a benign miss (404, deleted between list -// and fetch) is reported as ErrObjectNotFound so the caller can skip it, while a -// real failure (500 — e.g. a permission problem) surfaces as a generic error so -// the caller does NOT mistake it for an empty result. -func TestFetchDistinguishesFailures(t *testing.T) { - allowDial(t) - - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - switch r.URL.Path { - case "/b/ok.txt": - w.Header().Set("Content-Type", "text/plain") - w.Write([]byte("fine")) - case "/b/boom.txt": - http.Error(w, "kaboom", http.StatusInternalServerError) - default: - http.Error(w, "not found", http.StatusNotFound) - } - })) - defer srv.Close() - - c := New(Config{ - Endpoint: srv.URL, - Region: "us-east-1", - Bucket: "b", - AccessKeyID: "AKIAEXAMPLE", - }, "secret") - - ok, err := c.Fetch(context.Background(), "ok.txt") - if err != nil || string(ok.Body) != "fine" { - t.Fatalf("ok.txt: body=%q err=%v", ok.Body, err) - } - if _, err := c.Fetch(context.Background(), "boom.txt"); err == nil || errors.Is(err, ErrObjectNotFound) { - t.Fatalf("a 500 must be a real error (not ErrObjectNotFound), got %v", err) - } - if _, err := c.Fetch(context.Background(), "gone.txt"); !errors.Is(err, ErrObjectNotFound) { - t.Fatalf("a 404 must be ErrObjectNotFound, got %v", err) - } -} - -// TestFetchReportsTooLarge proves an object over the transport ceiling surfaces -// as ErrObjectTooLarge rather than being silently discarded. -func TestFetchReportsTooLarge(t *testing.T) { - allowDial(t) - - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - w.Write(make([]byte, maxResponseBytes+1)) - })) - defer srv.Close() - - c := New(Config{ - Endpoint: srv.URL, - Region: "us-east-1", - Bucket: "b", - AccessKeyID: "AKIAEXAMPLE", - }, "secret") - if _, err := c.Fetch(context.Background(), "big.bin"); !errors.Is(err, ErrObjectTooLarge) { - t.Fatalf("oversized object: err = %v, want ErrObjectTooLarge", err) - } -} - -// TestSSRFGuardBlocksLoopback confirms the dial guard is wired by default (no -// override): a loopback endpoint is refused. -func TestSSRFGuardBlocksLoopback(t *testing.T) { - c := New(Config{ - Endpoint: "http://127.0.0.1:1", - Bucket: "b", - Region: "us-east-1", - AccessKeyID: "AKIAEXAMPLE", - }, "secret") - if _, err := c.List(context.Background()); err == nil { - t.Fatal("expected an error listing a loopback endpoint, got nil") - } -} - -// TestListReportsATruncatedListingIncomplete pins the completeness signal a -// mirror deletes by: a page the store marks IsTruncated, or one the object cap -// cut, is never complete, while an untruncated page within the cap is. -func TestListReportsATruncatedListingIncomplete(t *testing.T) { - allowDial(t) - truncated := "false" - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - w.Header().Set("Content-Type", "application/xml") - w.Write([]byte(` -b` + truncated + ` - docs/a.txt"a" - docs/b.txt"b" - docs/c.txt"c" -`)) - })) - defer srv.Close() - client := func(max int) *Client { - return New(Config{Endpoint: srv.URL, Region: "us-east-1", Bucket: "b", Prefix: "docs/", AccessKeyID: "AK", MaxObjects: max}, "secret") - } - for _, tc := range []struct { - name string - truncated string - max int - entries int - complete bool - }{ - {"whole listing within the cap", "false", 3, 3, true}, - {"store truncated the page", "true", 10, 3, false}, - {"object cap cut the page", "false", 2, 2, false}, - } { - t.Run(tc.name, func(t *testing.T) { - truncated = tc.truncated - listing, err := client(tc.max).List(context.Background()) - if err != nil { - t.Fatal(err) - } - if len(listing.Entries) != tc.entries || listing.Complete != tc.complete { - t.Fatalf("entries=%d complete=%v, want %d/%v", len(listing.Entries), listing.Complete, tc.entries, tc.complete) - } - }) - } -} diff --git a/module/services/accounts/code/pkg/datasource/objectstore/connector.go b/module/services/accounts/code/pkg/datasource/objectstore/connector.go new file mode 100644 index 000000000..e067eebc9 --- /dev/null +++ b/module/services/accounts/code/pkg/datasource/objectstore/connector.go @@ -0,0 +1,402 @@ +package objectstore + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "errors" + "fmt" + "io" + "sort" + "strings" + "time" + + "accounts/pkg/datasource/connector" + accountsv1 "accounts/pkg/gen/saas/accounts/v1" +) + +// The object-storage files connector, on the envelope. An object store has no +// "what changed since": it answers only "what is there now". So the version of +// a source is a digest of its listing (key → ETag → size), the host retains the +// listing each version names (Manifests), and Changes diffs the current listing +// against the retained one. The listing and the reads go through a Bucket, which +// is the composed object-storage service under the source's binding +// (codefly-dev/service-object-storage#108): this connector signs nothing, dials +// nothing and guards nothing itself. + +// ConnectorKey is the provider key object-storage sources are stored under. +const ConnectorKey = "upload" + +// Budget limits. An object past MaxItemBytes is refused, never truncated; +// MaxObjectBytes is the cap the object-storage service puts on a bound read +// (its Capabilities' max_object_bytes), so an object over it is refused here +// from its listed size rather than by the service mid-read. +const ( + MaxObjectsPerBatch = 1000 + MaxBatchBytes = 64 << 20 + MaxObjectBytes = 5 << 20 +) + +// estimatedThrottleWindow is the reset a throttled call is given when the store +// does not say when its limit resets. +const estimatedThrottleWindow = time.Minute + +// SourceConfig is where one source reads: its store, bucket and key prefix, and +// the Scope of what under that prefix one sync takes. The credential is not +// here; the host hands it to the Bucket per call. +type SourceConfig struct { + Endpoint string + Region string + Bucket string + Prefix string + Scope Scope +} + +// binding names the configuration a version was listed under, so a version of +// one bucket or prefix is never diffed against, or read from, another. +func (c SourceConfig) binding() string { + sum := sha256.Sum256([]byte(c.Endpoint + "\x00" + c.Region + "\x00" + c.Bucket + "\x00" + c.Prefix)) + return hex.EncodeToString(sum[:8]) +} + +// Item is one object of a listing: its full key (the item's stable id), its +// ETag (the item's version) and its exact size in bytes. +type Item struct { + Key string `json:"key"` + ETag string `json:"etag"` + Size int64 `json:"size"` +} + +// ErrListingIncomplete: the store could not list the whole prefix (it truncated +// the listing, or the listing passed the store's cap). A partial listing is never +// a version: it would read every unlisted object as deleted. +var ErrListingIncomplete = errors.New("objectstore: the store did not list the whole prefix") + +// ErrObjectChanged: an object named at an ETag no longer has it (it was +// rewritten or deleted after it was listed). +var ErrObjectChanged = errors.New("objectstore: the object is no longer at the listed version") + +// Bucket is the store as the connector reads it, for one source. It reports a +// throttle as *connector.RateLimitedError and a refused credential as +// connector.ErrCredentialRefused. +type Bucket interface { + // List returns every object under the source's prefix, or + // ErrListingIncomplete; never a partial listing. + List(ctx context.Context) ([]Item, error) + // Read streams the named objects in one call, in order, each only at its + // listed ETag (ErrObjectChanged otherwise), with a reader of exactly its size. + Read(ctx context.Context, items []Item, visit func(item Item, contentType string, content io.Reader) error) error +} + +// BucketFunc resolves the Bucket of one source, per call: the host decrypts the +// source's credential and binds it, and the connector never sees it. +type BucketFunc func(ctx context.Context, src connector.Source) (Bucket, error) + +// Manifests is the host's retained listings, per source and version. Put is +// idempotent; Get reports false for a version that was never retained or has +// since been pruned. +type Manifests interface { + Put(ctx context.Context, src connector.Source, version string, items []Item) error + Get(ctx context.Context, src connector.Source, version string) ([]Item, bool, error) +} + +// FilesConnector is the object-storage connector. +type FilesConnector struct { + bucket BucketFunc + manifests Manifests + budget connector.Budget + now func() time.Time +} + +// ConnectorOption configures a FilesConnector. +type ConnectorOption func(*FilesConnector) + +// WithMaxItemBytes lowers the per-object limit, for a test that must script an +// over-limit object cheaply. It never raises it past MaxObjectBytes. +func WithMaxItemBytes(n int64) ConnectorOption { + return func(c *FilesConnector) { + if n > 0 && n < c.budget.MaxItemBytes { + c.budget.MaxItemBytes = n + } + } +} + +// NewFilesConnector returns the object-storage connector over bucket, retaining +// listings in manifests. +func NewFilesConnector(bucket BucketFunc, manifests Manifests, opts ...ConnectorOption) *FilesConnector { + c := &FilesConnector{ + bucket: bucket, + manifests: manifests, + budget: connector.Budget{ + MaxItemsPerCall: MaxObjectsPerBatch, MaxBytesPerCall: MaxBatchBytes, MaxItemBytes: MaxObjectBytes, + // One operation is one listing or one batched read. A store meters + // requests per second, far above this; the quota is what the host + // spends on a tenant's bucket, with a fifth kept for a person's sync. + OperationsPerWindow: 1200, Window: time.Hour, BackgroundSharePercent: 80, + }, + now: time.Now, + } + for _, opt := range opts { + opt(c) + } + return c +} + +// Descriptor is the connector's catalog entry. +func (c *FilesConnector) Descriptor() connector.Descriptor { + return connector.Descriptor{ + Key: ConnectorKey, + DisplayName: "Object storage", + Description: "A bucket in an object store, read through the object-storage service; the objects under a prefix that pass its key filter are pulled on sync. A source is one collection with one set of readers, so a bucket shared by several audiences is several sources.", + Interface: connector.InterfaceFiles, + ConfigFields: []connector.ConfigField{ + {Key: "endpoint", DisplayName: "Endpoint", Help: "The store's origin, http(s) with no path; the object-storage service refuses one that is not on the public internet.", Required: true}, + {Key: "region", DisplayName: "Region", Help: "The bucket's region, e.g. us-east-1.", Required: true}, + {Key: "bucket", DisplayName: "Bucket", Required: true}, + {Key: "prefix", DisplayName: "Prefix", Help: "Key prefix to read under; empty means the whole bucket.", Required: false}, + {Key: "include_patterns", DisplayName: "Include patterns", Help: "RE2 expressions over the key after the prefix; empty includes everything.", Required: false}, + {Key: "exclude_patterns", DisplayName: "Exclude patterns", Help: "RE2 expressions over the key after the prefix; an excluded key is never read.", Required: false}, + {Key: "file_extensions", DisplayName: "File types", Help: "Suffixes to keep, as for a GitHub source; empty keeps every type.", Required: false}, + {Key: "access_key_id", DisplayName: "Access key ID", Required: true}, + }, + CredentialModes: []connector.CredentialMode{connector.CredentialStaticSecret}, + // A bucket has no per-object access list: every object is readable by + // whoever may read the source's collection. One collection per prefix. + Readers: connector.ReadersSourceScoped, + Budget: c.budget, + Conformant: true, + } +} + +func (c *FilesConnector) open(ctx context.Context, src connector.Source) (SourceConfig, Bucket, error) { + cfg, ok := src.Config.(SourceConfig) + if !ok || cfg.Bucket == "" { + return SourceConfig{}, nil, fmt.Errorf("objectstore connector: source %s has no object-storage configuration", src.ID) + } + if err := ctx.Err(); err != nil { + return SourceConfig{}, nil, err + } + bucket, err := c.bucket(ctx, src) + if err != nil { + return SourceConfig{}, nil, c.envelopeError(err) + } + return cfg, bucket, nil +} + +// list reads the current listing within the source's Scope, and its version. +// Folder markers (keys ending in "/") are not files and are left out. A listing +// the Scope's cap cuts is ErrListingIncomplete: the version names every object +// in scope or none. +func (c *FilesConnector) list(ctx context.Context, cfg SourceConfig, bucket Bucket) (string, map[string]Item, error) { + items, err := bucket.List(ctx) + if err != nil { + return "", nil, c.envelopeError(err) + } + files := make([]Item, 0, len(items)) + for _, it := range items { + if !strings.HasSuffix(it.Key, "/") { + files = append(files, it) + } + } + kept, whole := Window(cfg.Scope, files) + if !whole { + return "", nil, fmt.Errorf("%w: more than %d objects are in scope", ErrListingIncomplete, cfg.Scope.limit()) + } + listed := make(map[string]Item, len(kept)) + for _, it := range kept { + listed[it.Key] = it + } + return versionOf(cfg, listed), listed, nil +} + +// versionOf is the listing's digest under its binding: the binding's digest, +// then the digest of every key, ETag and size in key order. +func versionOf(cfg SourceConfig, listed map[string]Item) string { + h := sha256.New() + for _, key := range sortedKeys(listed) { + it := listed[key] + _, _ = fmt.Fprintf(h, "%s\x00%s\x00%d\n", it.Key, it.ETag, it.Size) // a hash never fails a write + } + return cfg.binding() + "." + hex.EncodeToString(h.Sum(nil)) +} + +func sortedKeys(m map[string]Item) []string { + keys := make([]string, 0, len(m)) + for k := range m { + keys = append(keys, k) + } + sort.Strings(keys) + return keys +} + +// Version lists the prefix and digests it. An object store answers no cheaper +// question than its listing, so this costs one listing call. +func (c *FilesConnector) Version(ctx context.Context, src connector.Source) (string, error) { + cfg, bucket, err := c.open(ctx, src) + if err != nil { + return "", err + } + version, _, err := c.list(ctx, cfg, bucket) + return version, err +} + +// Changes diffs the current listing against the retained listing from names, +// and retains the current one under its version. An empty from is a complete +// snapshot. A from that was listed under another binding, or whose listing is +// no longer retained, requires a resync. +// +// An object store has no rename: a key that disappeared and a key that appeared +// with the same ETag and size are the same content moved, and are reported as +// one MOVED change, paired in key order. +func (c *FilesConnector) Changes(ctx context.Context, src connector.Source, from string) (connector.ChangeSet, error) { + cfg, bucket, err := c.open(ctx, src) + if err != nil { + return connector.ChangeSet{}, err + } + var previous map[string]Item + if from != "" { + if !strings.HasPrefix(from, cfg.binding()+".") { + return connector.ChangeSet{}, connector.ErrResyncRequired + } + items, ok, err := c.manifests.Get(ctx, src, from) + if err != nil { + return connector.ChangeSet{}, err + } + if !ok { + return connector.ChangeSet{}, connector.ErrResyncRequired + } + previous = make(map[string]Item, len(items)) + for _, it := range items { + previous[it.Key] = it + } + } + version, listed, err := c.list(ctx, cfg, bucket) + if err != nil { + return connector.ChangeSet{}, err + } + set := connector.ChangeSet{SourceID: src.ID, From: from, To: version, Complete: from == ""} + if version == from { + return set, nil + } + current := make([]Item, 0, len(listed)) + for _, key := range sortedKeys(listed) { + current = append(current, listed[key]) + } + if err := c.manifests.Put(ctx, src, version, current); err != nil { + return connector.ChangeSet{}, err + } + readers := connector.ApplySourcePolicy(src, connector.SourceScoped()) + change := func(it Item, kind connector.ChangeKind) connector.Change { + return connector.Change{Key: connector.KeyFor(src, it.Key), Kind: kind, ItemVersion: it.ETag, Size: it.Size, Locator: it.Key, Readers: readers} + } + var added, deleted []Item + for _, it := range current { + prev, ok := previous[it.Key] + switch { + case !ok: + added = append(added, it) + case prev.ETag != it.ETag || prev.Size != it.Size: + set.Changes = append(set.Changes, change(it, connector.ChangeModified)) + } + } + for _, key := range sortedKeys(previous) { + if _, ok := listed[key]; !ok { + deleted = append(deleted, previous[key]) + } + } + moved := make(map[string]bool) + for _, it := range added { + kind, prevKey := connector.ChangeAdded, "" + for _, gone := range deleted { + if !moved[gone.Key] && gone.ETag == it.ETag && gone.Size == it.Size { + moved[gone.Key], kind, prevKey = true, connector.ChangeMoved, gone.Key + break + } + } + ch := change(it, kind) + ch.PreviousItemID = prevKey + set.Changes = append(set.Changes, ch) + } + for _, gone := range deleted { + if !moved[gone.Key] { + set.Changes = append(set.Changes, connector.Change{Key: connector.KeyFor(src, gone.Key), Kind: connector.ChangeDeleted, Locator: gone.Key}) + } + } + return set, nil +} + +// FetchFiles reads the named objects as the retained listing of version names +// them, in one batched read. An object rewritten or deleted since that listing +// is not at that version any more, and is ErrItemNotFound. +func (c *FilesConnector) FetchFiles(ctx context.Context, src connector.Source, version string, refs []connector.FileRef, visit func(connector.File, io.Reader) error) error { + if len(refs) == 0 || len(refs) > c.budget.MaxItemsPerCall { + return fmt.Errorf("%w: a batch names 1 to %d objects", connector.ErrBatchTooLarge, c.budget.MaxItemsPerCall) + } + seen := make(map[string]bool, len(refs)) + for _, ref := range refs { + if ref.ItemID == "" || seen[ref.ItemID] { + return errors.New("objectstore connector: every object of a batch names a distinct item") + } + seen[ref.ItemID] = true + } + cfg, bucket, err := c.open(ctx, src) + if err != nil { + return err + } + if !strings.HasPrefix(version, cfg.binding()+".") { + return connector.ErrVersionNotFound + } + retained, ok, err := c.manifests.Get(ctx, src, version) + if err != nil { + return err + } + if !ok { + return connector.ErrVersionNotFound + } + byKey := make(map[string]Item, len(retained)) + for _, it := range retained { + byKey[it.Key] = it + } + items := make([]Item, len(refs)) + var total int64 + for i, ref := range refs { + it, ok := byKey[ref.ItemID] + if !ok || (ref.ItemVersion != "" && ref.ItemVersion != it.ETag) { + return connector.ErrItemNotFound + } + if it.Size > c.budget.MaxItemBytes { + return fmt.Errorf("%w: an object is over %d bytes", connector.ErrItemTooLarge, c.budget.MaxItemBytes) + } + items[i], total = it, total+it.Size + } + if total > c.budget.MaxBytesPerCall { + return fmt.Errorf("%w: the batch is %d bytes, over %d; request fewer objects", connector.ErrBatchTooLarge, total, c.budget.MaxBytesPerCall) + } + readers := connector.ApplySourcePolicy(src, connector.SourceScoped()) + err = bucket.Read(ctx, items, func(it Item, contentType string, content io.Reader) error { + return visit(connector.File{ + Provenance: &accountsv1.DatasourceProvenance{ + SourceId: src.ID, OrgId: src.OrgID, BoundaryNodeId: src.BoundaryNodeID, + Version: version, ItemId: it.Key, ItemVersion: it.ETag, + }, + Locator: it.Key, + ContentType: contentType, + Size: it.Size, + Readers: readers, + }, content) + }) + if errors.Is(err, ErrObjectChanged) { + return fmt.Errorf("%w: %w", connector.ErrItemNotFound, err) + } + return c.envelopeError(err) +} + +// envelopeError gives a throttle the store did not time a reset, keeping every +// other error, typed by the Bucket, as it is. +func (c *FilesConnector) envelopeError(err error) error { + var limited *connector.RateLimitedError + if errors.As(err, &limited) && limited.ResetAt.IsZero() { + limited.ResetAt, limited.Estimated = c.now().Add(estimatedThrottleWindow), true + } + return err +} diff --git a/module/services/accounts/code/pkg/datasource/objectstore/connector_conformance_test.go b/module/services/accounts/code/pkg/datasource/objectstore/connector_conformance_test.go new file mode 100644 index 000000000..7ff4749e4 --- /dev/null +++ b/module/services/accounts/code/pkg/datasource/objectstore/connector_conformance_test.go @@ -0,0 +1,189 @@ +package objectstore + +import ( + "context" + "crypto/md5" + "encoding/hex" + "errors" + "fmt" + "io" + "sync" + "testing" + "time" + + "accounts/pkg/datasource/connector" + "accounts/pkg/datasource/connector/connectortest" +) + +// TestFilesConnectorConformance runs the datasource conformance suite against +// the object-storage connector reading through ServiceBucket, the host's client +// of the object-storage service, over an in-process fake of the service's gRPC +// API: the binding, the paged listing and the batched read on the wire, a +// sentinel credential the store checks, and every RPC counted — so clause 8's +// bound is measured on the calls the host actually makes. +func TestFilesConnectorConformance(t *testing.T) { + connectortest.RunFiles(t, newConformanceFixture) +} + +const conformanceCredential = "sentinel-secret-key-4c1d" +const conformanceMaxItemBytes = 4 << 10 + +// The suite lists 120 objects in one snapshot: the source is configured at the +// ceiling, past the 100 an unconfigured source takes. Its keys are written +// without a prefix, so the source reads the whole bucket. +var conformanceConfig = SourceConfig{Endpoint: "https://store.example.com", Region: "us-east-1", Bucket: "acme-docs", Scope: Scope{MaxObjects: maxObjectsCeiling}} + +type conformanceFixture struct { + store *fakeService + manifests *memoryManifests + conn *FilesConnector + sources int +} + +func newConformanceFixture(t *testing.T) connectortest.FilesFixture { + f := &conformanceFixture{store: &fakeService{objects: map[string]string{}, pageSize: 50, secret: conformanceCredential}, manifests: &memoryManifests{}} + client := serve(t, f.store) + f.conn = NewFilesConnector(func(_ context.Context, src connector.Source) (Bucket, error) { + // The host resolves the credential per call; here it is the sentinel. + cfg, _ := src.Config.(SourceConfig) + return NewServiceBucket(client, cfg, "conformance-key", conformanceCredential), nil + }, f.manifests, WithMaxItemBytes(conformanceMaxItemBytes)) + return f +} + +func (f *conformanceFixture) Connector() connector.FilesConnector { return f.conn } + +func (f *conformanceFixture) Source(t *testing.T, org string) connector.Source { + f.sources++ + return connector.Source{ + ID: fmt.Sprintf("aaaaaaaa-0000-0000-0000-%012d", f.sources), + OrgID: org, + BoundaryNodeID: fmt.Sprintf("bbbbbbbb-0000-0000-0000-%012d", f.sources), + Config: conformanceConfig, + } +} + +// Apply writes the provider's next state. A store has no rename: a move is the +// object written at its new key and removed from its old one. +func (f *conformanceFixture) Apply(t *testing.T, puts map[string]string, deletes []string, moves map[string]string) { + f.store.mu.Lock() + defer f.store.mu.Unlock() + for key, content := range puts { + f.store.objects[key] = content + } + for _, key := range deletes { + delete(f.store.objects, key) + } + for from, to := range moves { + f.store.objects[to] = f.store.objects[from] + delete(f.store.objects, from) + } +} + +// Rewrite: a bucket has no history to rewrite. What makes a version +// undiffable is that the listing it names is no longer retained, so the +// fixture drops every retained listing. +func (f *conformanceFixture) Rewrite(t *testing.T) { f.manifests.forget() } + +func (f *conformanceFixture) ProviderCalls() int64 { return f.store.calls.Load() } + +func (f *conformanceFixture) RateLimit(t *testing.T) func() { + f.store.throttled.Store(true) + return func() { f.store.throttled.Store(false) } +} + +func (f *conformanceFixture) Credential() string { return conformanceCredential } + +func (f *conformanceFixture) CredentialPresented() bool { return f.store.authenticated.Load() } + +func (f *conformanceFixture) MaxItemBytes() int64 { return conformanceMaxItemBytes } + +func etagOf(content string) string { + sum := md5.Sum([]byte(content)) + return `"` + hex.EncodeToString(sum[:]) + `"` +} + +// memoryManifests retains every listing until forget. +type memoryManifests struct { + mu sync.Mutex + kept map[string][]Item +} + +func (m *memoryManifests) Put(_ context.Context, src connector.Source, version string, items []Item) error { + m.mu.Lock() + defer m.mu.Unlock() + if m.kept == nil { + m.kept = map[string][]Item{} + } + m.kept[src.ID+"/"+version] = append([]Item(nil), items...) + return nil +} + +func (m *memoryManifests) Get(_ context.Context, src connector.Source, version string) ([]Item, bool, error) { + m.mu.Lock() + defer m.mu.Unlock() + items, ok := m.kept[src.ID+"/"+version] + return items, ok, nil +} + +func (m *memoryManifests) forget() { + m.mu.Lock() + defer m.mu.Unlock() + m.kept = nil +} + +// TestChangesResyncAcrossABinding holds a version to the bucket and prefix it +// was listed under: after the source is pointed elsewhere, its old version is +// neither diffed against nor read from. +func TestChangesResyncAcrossABinding(t *testing.T) { + f := newConformanceFixture(t).(*conformanceFixture) + f.Apply(t, map[string]string{"kb/a.md": "alpha"}, nil, nil) + src := f.Source(t, "11111111-1111-1111-1111-111111111111") + ctx := context.Background() + a, err := f.conn.Changes(ctx, src, "") + if err != nil { + t.Fatal(err) + } + moved := conformanceConfig + moved.Prefix = "other/" + src.Config = moved + if _, err := f.conn.Changes(ctx, src, a.To); !errors.Is(err, connector.ErrResyncRequired) { + t.Fatalf("changes from a version of another prefix = %v, want ErrResyncRequired", err) + } + err = f.conn.FetchFiles(ctx, src, a.To, []connector.FileRef{{ItemID: "kb/a.md"}}, func(connector.File, io.Reader) error { return nil }) + if !errors.Is(err, connector.ErrVersionNotFound) { + t.Fatalf("fetching a version of another prefix = %v, want ErrVersionNotFound", err) + } +} + +// TestFetchRefusesARewrittenObject: an object rewritten after it was listed is +// not at the listed version any more, so a fetch at that version refuses it +// rather than serve content the version never had. +func TestFetchRefusesARewrittenObject(t *testing.T) { + f := newConformanceFixture(t).(*conformanceFixture) + f.Apply(t, map[string]string{"kb/a.md": "alpha"}, nil, nil) + src := f.Source(t, "11111111-1111-1111-1111-111111111111") + ctx := context.Background() + a, err := f.conn.Changes(ctx, src, "") + if err != nil { + t.Fatal(err) + } + f.Apply(t, map[string]string{"kb/a.md": "alpha, rewritten"}, nil, nil) + err = f.conn.FetchFiles(ctx, src, a.To, []connector.FileRef{{ItemID: "kb/a.md"}}, func(connector.File, io.Reader) error { return nil }) + if !errors.Is(err, connector.ErrItemNotFound) { + t.Fatalf("fetching an object rewritten since its listing = %v, want ErrItemNotFound", err) + } +} + +// TestRateLimitIsEstimatedWhenTheStoreGivesNoReset: a store's throttle names +// no reset, so the connector gives it one rather than a zero time. +func TestRateLimitIsEstimatedWhenTheStoreGivesNoReset(t *testing.T) { + f := newConformanceFixture(t).(*conformanceFixture) + clear := f.RateLimit(t) + defer clear() + _, err := f.conn.Changes(context.Background(), f.Source(t, "11111111-1111-1111-1111-111111111111"), "") + var limited *connector.RateLimitedError + if !errors.As(err, &limited) || !limited.Estimated || !limited.ResetAt.After(time.Now()) { + t.Fatalf("a throttle without a reset = %v, want an estimated reset in the future", err) + } +} diff --git a/module/services/accounts/code/pkg/datasource/objectstore/guard_test.go b/module/services/accounts/code/pkg/datasource/objectstore/guard_test.go deleted file mode 100644 index 8b45e5833..000000000 --- a/module/services/accounts/code/pkg/datasource/objectstore/guard_test.go +++ /dev/null @@ -1,133 +0,0 @@ -package objectstore - -import ( - "context" - "errors" - "net/http" - "net/http/httptest" - "reflect" - "strings" - "sync/atomic" - "testing" - "time" - - "accounts/pkg/auth" - "accounts/pkg/auth/egresstest" -) - -func TestGuardDial_RefusesEveryNonPublicRange(t *testing.T) { - for _, tc := range egresstest.Refused { - t.Run(tc.Name, func(t *testing.T) { - if err := guardDial(tc.DialAddress())("tcp", tc.DialAddress(), nil); !errors.Is(err, ErrBlockedAddress) { - t.Fatalf("guardDial(%q) = %v, want ErrBlockedAddress", tc.DialAddress(), err) - } - }) - } - for _, tc := range egresstest.Unclassifiable { - t.Run(tc.Name, func(t *testing.T) { - if err := guardDial(tc.Address)("tcp", tc.Address, nil); !errors.Is(err, ErrBlockedAddress) { - t.Fatalf("guardDial(%q) = %v, want ErrBlockedAddress", tc.Address, err) - } - }) - } -} - -func TestGuardDial_AdmitsPublicAddresses(t *testing.T) { - for _, tc := range egresstest.Admitted { - t.Run(tc.Name, func(t *testing.T) { - if err := guardDial(tc.DialAddress())("tcp", tc.DialAddress(), nil); err != nil { - t.Fatalf("guardDial(%q) = %v, want admitted", tc.DialAddress(), err) - } - }) - } -} - -// Production wires guardDial as the dial hook, and the listing request and every object request runs it on -// the address being dialed. The hook is wrapped in a recorder that notes the -// guard's verdict and then stops the dial, and the only target is a local -// listener, so the test never dials anything but this process. -func TestEveryOutboundDialIsGuarded(t *testing.T) { - var reached atomic.Int64 - live := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, _ *http.Request) { - reached.Add(1) - })) - defer live.Close() - listener := strings.TrimPrefix(live.URL, "http://") - origin := live.URL - - if reflect.ValueOf(dialGuard).Pointer() != reflect.ValueOf(guardDial).Pointer() { - t.Fatal("the production dial hook is not guardDial") - } - recorder := &egresstest.Recorder{} - saved := dialGuard - dialGuard = func(requested string) auth.DialControl { return recorder.Wrap(guardDial(requested)) } - t.Cleanup(func() { dialGuard = saved }) - - paths := []struct { - name string - call func(ctx context.Context) error - }{ - {"listing dial (List)", func(ctx context.Context) error { - _, err := newGuardedClient(origin).List(ctx) - return err - }}, - {"object dial (Fetch)", func(ctx context.Context) error { - _, err := newGuardedClient(origin).Fetch(ctx, "docs/page.md") - return err - }}, - } - for _, path := range paths { - t.Run(path.name, func(t *testing.T) { - recorder.Take() - ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second) - defer cancel() - err := path.call(ctx) - got := recorder.Take() - if len(got) == 0 { - t.Fatal("the dial did not run the guard") - } - for _, v := range got { - if v.Address != listener { - t.Fatalf("the guard judged %q, want the address being dialed %q", v.Address, listener) - } - if !errors.Is(v.Err, ErrBlockedAddress) { - t.Fatalf("guard(%q) = %v, want ErrBlockedAddress", v.Address, v.Err) - } - } - if !errors.Is(err, ErrBlockedAddress) { - t.Fatalf("call error = %v, want ErrBlockedAddress", err) - } - }) - } - if n := reached.Load(); n != 0 { - t.Fatalf("live listener was reached %d times", n) - } -} - -func newGuardedClient(endpoint string) *Client { - return New(Config{Endpoint: endpoint, Bucket: "example-bucket", Region: "us-east-1", AccessKeyID: "AKIAEXAMPLE"}, "example-secret") -} - -// Every outbound path applies the egress rule's mesh admission end to end: a -// declared host a meshed cell resolves into its mesh egress range is dialled, -// and every case around it is refused (egresstest.MeshCases). DNS is a fake -// and an admitted dial is stopped before it connects. -func TestEveryOutboundPathAppliesTheMeshEgressRule(t *testing.T) { - egresstest.RunMeshCases(t, []egresstest.MeshPath{ - {Name: "listing dial (List)", Call: func(ctx context.Context, origin string) error { - _, err := newGuardedClient(origin).List(ctx) - return err - }}, - {Name: "object dial (Fetch)", Call: func(ctx context.Context, origin string) error { - _, err := newGuardedClient(origin).Fetch(ctx, "docs/page.md") - return err - }}, - }, ErrBlockedAddress, func(t *testing.T, c egresstest.MeshCase) *egresstest.Recorder { - recorder := &egresstest.Recorder{} - savedGuard, savedResolver := dialGuard, dialResolver - dialGuard = func(requested string) auth.DialControl { return recorder.Wrap(guardDial(requested)) } - dialResolver = c.Resolver() - t.Cleanup(func() { dialGuard, dialResolver = savedGuard, savedResolver }) - return recorder - }) -} diff --git a/module/services/accounts/code/pkg/datasource/objectstore/scope.go b/module/services/accounts/code/pkg/datasource/objectstore/scope.go new file mode 100644 index 000000000..ad360a18f --- /dev/null +++ b/module/services/accounts/code/pkg/datasource/objectstore/scope.go @@ -0,0 +1,50 @@ +package objectstore + +// defaultMaxObjects caps how many objects one sync takes when the source does +// not configure a limit; maxObjectsCeiling is the hard upper bound regardless of +// configuration. +const ( + defaultMaxObjects = 100 + maxObjectsCeiling = 1000 +) + +// Scope is which of a prefix's objects one sync takes: those whose key InScope +// admits — the host's key filter for the source, as a GitHub source's +// SourceConfig.InScope is its path scope — up to the object cap. It belongs to +// the connector, not to what lists the prefix: a listing source hands Window +// everything it listed, so replacing the listing source leaves the filter and +// the cap where they are. +type Scope struct { + InScope func(key string) bool // nil admits every key + MaxObjects int // 0 means the default cap +} + +func (s Scope) limit() int { + switch { + case s.MaxObjects <= 0: + return defaultMaxObjects + case s.MaxObjects > maxObjectsCeiling: + return maxObjectsCeiling + } + return s.MaxObjects +} + +// Window keeps the listed items whose key the scope admits, in listing order +// and at most the cap of them. An item it refuses is dropped before it is +// counted, so an excluded object never uses up the cap and is never fetched. +// whole is false when the cap cut an admitted item: a caller that reads +// absence as deletion must then not act on what it kept. +func Window(s Scope, items []Item) (kept []Item, whole bool) { + limit := s.limit() + kept = make([]Item, 0, min(len(items), limit)) + admitted := 0 + for _, item := range items { + if s.InScope != nil && !s.InScope(item.Key) { + continue + } + if admitted++; admitted <= limit { + kept = append(kept, item) + } + } + return kept, admitted <= limit +} diff --git a/module/services/accounts/code/pkg/datasource/objectstore/scope_test.go b/module/services/accounts/code/pkg/datasource/objectstore/scope_test.go new file mode 100644 index 000000000..30fa9599a --- /dev/null +++ b/module/services/accounts/code/pkg/datasource/objectstore/scope_test.go @@ -0,0 +1,43 @@ +package objectstore + +import ( + "slices" + "strings" + "testing" +) + +// TestWindow pins what a sync takes of a listing: the admitted keys in listing +// order, at most the cap, with an excluded key dropped before it is counted. +func TestWindow(t *testing.T) { + listed := []Item{{Key: "docs/a.md"}, {Key: "docs/drafts/b.md"}, {Key: "docs/c.md"}, {Key: "docs/d.pdf"}} + noDrafts := func(k string) bool { return !strings.HasPrefix(k, "docs/drafts/") } + for _, tc := range []struct { + name string + scope Scope + keys []string + whole bool + }{ + {"no filter, within the cap", Scope{MaxObjects: 4}, []string{"docs/a.md", "docs/drafts/b.md", "docs/c.md", "docs/d.pdf"}, true}, + {"the cap cut an admitted key", Scope{MaxObjects: 2}, []string{"docs/a.md", "docs/drafts/b.md"}, false}, + {"an excluded key never uses up the cap", Scope{InScope: noDrafts, MaxObjects: 3}, []string{"docs/a.md", "docs/c.md", "docs/d.pdf"}, true}, + {"the cap counts only admitted keys", Scope{InScope: noDrafts, MaxObjects: 2}, []string{"docs/a.md", "docs/c.md"}, false}, + {"no cap configured is the default", Scope{InScope: noDrafts}, []string{"docs/a.md", "docs/c.md", "docs/d.pdf"}, true}, + } { + t.Run(tc.name, func(t *testing.T) { + kept, whole := Window(tc.scope, listed) + var keys []string + for _, e := range kept { + keys = append(keys, e.Key) + } + if !slices.Equal(keys, tc.keys) || whole != tc.whole { + t.Fatalf("kept %v whole=%v, want %v whole=%v", keys, whole, tc.keys, tc.whole) + } + }) + } + if got := (Scope{MaxObjects: 5000}).limit(); got != maxObjectsCeiling { + t.Fatalf("a cap above the ceiling = %d, want %d", got, maxObjectsCeiling) + } + if got := (Scope{}).limit(); got != defaultMaxObjects { + t.Fatalf("no cap = %d, want %d", got, defaultMaxObjects) + } +} diff --git a/module/services/accounts/code/pkg/datasource/objectstore/service.go b/module/services/accounts/code/pkg/datasource/objectstore/service.go new file mode 100644 index 000000000..da8aac127 --- /dev/null +++ b/module/services/accounts/code/pkg/datasource/objectstore/service.go @@ -0,0 +1,350 @@ +package objectstore + +import ( + "bytes" + "context" + "errors" + "fmt" + "io" + "strings" + "time" + + storagev0 "github.com/codefly-dev/service-object-storage/gen/codefly/storage/v0" + "google.golang.org/genproto/googleapis/rpc/errdetails" + "google.golang.org/grpc" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/credentials/insecure" + "google.golang.org/grpc/status" + + "accounts/pkg/datasource/connector" +) + +// The Bucket the host reads through: the composed object-storage service +// (codefly-dev/service-object-storage), under a per-call Binding that names the +// source's store, bucket and prefix and carries its credential for that one +// call. The service owns the store protocol, every backend, the +// public-destination guard on each dial and the caps; the host keeps the +// source, the credential envelope and the retained listings. + +// maxListPages bounds one listing: a prefix whose listing runs past this many +// of the service's pages is not listed whole, and a partial listing is never a +// version. +const maxListPages = 20 + +// callTimeout bounds every call but a batched read, and readTimeout a batched +// read of up to MaxBatchBytes: a store that answers and then stalls must not +// hold a sync, and its worker's lease, for longer. Variables so a test can +// shorten them. +var ( + callTimeout = 30 * time.Second + readTimeout = 5 * time.Minute +) + +// ErrBindingNotHonoured: the service answered Capabilities under the binding +// without the caps a bound call reports, which is a release from before the +// binding existed — such a service ignores the binding and serves its own +// configured bucket, so nothing it lists or reads is taken for the source's — +// or without a batch size, which is one from before GetMany. +var ErrBindingNotHonoured = errors.New("objectstore: the object-storage service does not honour a per-call binding with a batched read; an older one would serve its own bucket") + +// errOutsideScope: the service listed what the source's literal prefix does +// not name, or more than its own page size. Nothing in such a listing is taken +// as the source's, since the closing listing would reconcile deletions against +// it. +var errOutsideScope = errors.New("objectstore: the object-storage service listed outside the source's scope") + +// ServiceBucket is one source's bucket as the object-storage service serves it. +type ServiceBucket struct { + client storagev0.ObjectStorageClient + binding *storagev0.Binding + prefix string // the source's literal key prefix + maxBatch int // objects per GetMany, once Capabilities under the binding answered + maxList int // objects per List page, likewise +} + +// NewServiceBucket binds cfg and the source's access key to calls on client. +// A source's prefix is a literal S3 key prefix — "/reports/" is not "reports/", +// "/" is not the whole bucket, and "reports-2026" matches "reports-2026-q1.pdf" +// — and the binding carries none: every listing sends it as the request's own +// prefix, every key travels whole, and the host refuses a listed key outside +// it. The host's scope therefore does not rest on how a service treats a +// binding's prefix (v0.0.19 keeps one literal; earlier releases normalized it +// as a folder), and the contract tests hold the pinned service to the request +// prefix being used as written. +func NewServiceBucket(client storagev0.ObjectStorageClient, cfg SourceConfig, accessKeyID, secretAccessKey string) *ServiceBucket { + return &ServiceBucket{ + client: client, + binding: &storagev0.Binding{ + // "s3" with an explicit endpoint is any S3-style store, addressed + // path-style; the source's endpoint is required, so this never + // silently means AWS. + Kind: "s3", + Endpoint: cfg.Endpoint, + Region: cfg.Region, + Bucket: cfg.Bucket, + Credential: &storagev0.Credential{Kind: &storagev0.Credential_AccessKey{ + AccessKey: &storagev0.AccessKey{Id: accessKeyID, Secret: secretAccessKey}, + }}, + }, + prefix: cfg.Prefix, + } +} + +// List pages through the prefix. A listing longer than maxListPages pages is +// ErrListingIncomplete; a key outside the literal prefix, or a page past the +// service's own page size, refuses the whole listing. +func (b *ServiceBucket) List(ctx context.Context) ([]Item, error) { + if err := b.checkBinding(ctx); err != nil { + return nil, err + } + var items []Item + token := "" + for page := 0; page < maxListPages; page++ { + res, err := b.listPage(ctx, token) + if err != nil { + return nil, storeError(err) + } + if len(res.GetObjects()) > b.maxList { + return nil, fmt.Errorf("%w: a page of %d objects, past its page size of %d", errOutsideScope, len(res.GetObjects()), b.maxList) + } + for _, o := range res.GetObjects() { + if !strings.HasPrefix(o.GetKey(), b.prefix) { + return nil, fmt.Errorf("%w: %q is not under %q", errOutsideScope, o.GetKey(), b.prefix) + } + items = append(items, Item{Key: o.GetKey(), ETag: o.GetEtag(), Size: o.GetSize()}) + } + if token = res.GetNextPageToken(); token == "" { + return items, nil + } + } + return nil, fmt.Errorf("%w: more than %d pages", ErrListingIncomplete, maxListPages) +} + +func (b *ServiceBucket) listPage(ctx context.Context, token string) (*storagev0.ListResult, error) { + ctx, cancel := context.WithTimeout(ctx, callTimeout) + defer cancel() + return b.client.List(ctx, &storagev0.ListRequest{Prefix: b.prefix, PageToken: token, Binding: b.binding}) +} + +// Read gets the batch at its listed ETags in one GetMany call per +// max_get_many_items objects (1000, which is the connector's batch), streamed +// back in request order: each object's header, then its bytes. An object +// rewritten or gone since its listing is ErrObjectChanged; one over the +// service's cap is the envelope's ErrItemTooLarge. +func (b *ServiceBucket) Read(ctx context.Context, items []Item, visit func(Item, string, io.Reader) error) error { + if err := b.checkBinding(ctx); err != nil { + return err + } + skipped := &TooLargeError{} + for start := 0; start < len(items); start += b.maxBatch { + if err := b.readBatch(ctx, items[start:min(start+b.maxBatch, len(items))], visit, skipped); err != nil { + return err + } + } + if len(skipped.Keys) > 0 { + return skipped + } + return nil +} + +// TooLargeError names the objects of a read the object-storage service refused +// as over its cap. Every other object of the read was visited: the service +// skips such an object and keeps streaming, and so does the host. It is the +// envelope's ErrItemTooLarge. +type TooLargeError struct{ Keys []string } + +func (e *TooLargeError) Error() string { + return fmt.Sprintf("objectstore: %d object(s) over the object-storage service's cap were skipped", len(e.Keys)) +} + +func (e *TooLargeError) Unwrap() error { return connector.ErrItemTooLarge } + +func (b *ServiceBucket) readBatch(ctx context.Context, items []Item, visit func(Item, string, io.Reader) error, skipped *TooLargeError) error { + req := &storagev0.GetManyRequest{Binding: b.binding, Items: make([]*storagev0.GetManyItem, len(items))} + for i, it := range items { + req.Items[i] = &storagev0.GetManyItem{Key: it.Key, IfMatch: it.ETag} + } + ctx, cancel := context.WithTimeout(ctx, readTimeout) + defer cancel() + stream, err := b.client.GetMany(ctx, req) + if err != nil { + return storeError(err) + } + var ( + current *Item + contentType string + content bytes.Buffer + answered int + ) + finish := func() error { + if current == nil { + return nil + } + it := *current + current = nil + if int64(content.Len()) != it.Size { + return fmt.Errorf("%w: %s", ErrObjectChanged, it.Key) + } + return visit(it, contentType, bytes.NewReader(content.Bytes())) + } + for { + msg, err := stream.Recv() + if errors.Is(err, io.EOF) { + break + } + if err != nil { + return storeError(err) + } + if data, ok := msg.GetKind().(*storagev0.GetManyResponse_Data); ok { + if current == nil || int64(content.Len()+len(data.Data)) > current.Size { + return fmt.Errorf("%w: the service sent bytes no listed object has", ErrObjectChanged) + } + content.Write(data.Data) + continue + } + if err := finish(); err != nil { + return err + } + header := msg.GetHeader() + if int(header.GetIndex()) != answered { + return fmt.Errorf("objectstore: the service answered object %d where %d was next", header.GetIndex(), answered) + } + it := items[answered] + answered++ + switch header.GetOutcome() { + case storagev0.GetManyOutcome_GET_MANY_OUTCOME_OK: + current, contentType = &it, header.GetInfo().GetContentType() + content.Reset() + case storagev0.GetManyOutcome_GET_MANY_OUTCOME_CHANGED, storagev0.GetManyOutcome_GET_MANY_OUTCOME_NOT_FOUND: + return fmt.Errorf("%w: %s", ErrObjectChanged, it.Key) + case storagev0.GetManyOutcome_GET_MANY_OUTCOME_TOO_LARGE: + skipped.Keys = append(skipped.Keys, it.Key) + default: + return fmt.Errorf("objectstore: the service answered object %s with outcome %v", it.Key, header.GetOutcome()) + } + } + if err := finish(); err != nil { + return err + } + if answered != len(items) { + return fmt.Errorf("%w: the service answered %d of %d objects", ErrObjectChanged, answered, len(items)) + } + return nil +} + +// checkBinding asks the service, once per Bucket, what a call under the binding +// honours. A service that knows the binding reports the bound read's caps; one +// that does not drops the unknown field, answers for its configured bucket, +// and reports none — and would answer List and Get from that bucket too. A +// service that reports no batch size predates the batched read, which this +// client needs. +func (b *ServiceBucket) checkBinding(ctx context.Context) error { + if b.maxBatch > 0 { + return nil + } + ctx, cancel := context.WithTimeout(ctx, callTimeout) + defer cancel() + caps, err := b.client.Capabilities(ctx, &storagev0.CapabilitiesRequest{Binding: b.binding}) + if err != nil { + return storeError(err) + } + if caps.GetMaxObjectBytes() <= 0 || caps.GetMaxListLimit() <= 0 || caps.GetMaxGetManyItems() <= 0 { + return ErrBindingNotHonoured + } + b.maxBatch, b.maxList = int(caps.GetMaxGetManyItems()), int(caps.GetMaxListLimit()) + return nil +} + +// errorDomain is the ErrorInfo domain the object-storage service names every +// refusal's reason under (codefly-dev/service-object-storage v0.0.19). +const errorDomain = "object-storage.codefly.dev" + +// ErrDestinationRefused: the object-storage service's guard refused the +// source's endpoint — not on the public internet, nor a composed dependency. +// It is the source's configuration to fix; the credential was never tried. +var ErrDestinationRefused = errors.New("objectstore: the object-storage service refused the source's endpoint") + +// storeError types the service's status for the envelope by the reason the +// service names, not by its code: one code covers refusals that call for +// opposite actions. A too-long listing or response is not a throttle — it +// fails the listing (ErrListingIncomplete) rather than blocking the credential +// and retrying forever — and a guard refusal is not a refused credential. A +// throttle carries the store's RetryInfo. A status with no reason keeps the +// code's reading: PERMISSION_DENIED a refused credential, RESOURCE_EXHAUSTED a +// throttle. +func storeError(err error) error { + st := status.Convert(err) + reason := "" + var retry *errdetails.RetryInfo + for _, d := range st.Details() { + switch d := d.(type) { + case *errdetails.ErrorInfo: + if d.GetDomain() == errorDomain { + reason = d.GetReason() + } + case *errdetails.RetryInfo: + retry = d + } + } + if reason == "LISTING_INCOMPLETE" { + // The store marked its listing truncated with no way to continue. + return fmt.Errorf("%w: %w", ErrListingIncomplete, err) + } + switch st.Code() { + case codes.PermissionDenied: + if reason == "DESTINATION_REFUSED" { + return fmt.Errorf("%w: %w", ErrDestinationRefused, err) + } + return fmt.Errorf("%w: %w", connector.ErrCredentialRefused, err) + case codes.ResourceExhausted: + switch reason { + case "RESPONSE_TOO_LARGE", "LISTING_TOO_LONG": + return fmt.Errorf("%w: %w", ErrListingIncomplete, err) + case "OBJECT_TOO_LARGE": + return fmt.Errorf("%w: %w", connector.ErrItemTooLarge, err) + } + limited := &connector.RateLimitedError{Scope: "credential", Cause: err} + if retry != nil && retry.GetRetryDelay() != nil { + limited.ResetAt = time.Now().Add(retry.GetRetryDelay().AsDuration()) + } + return limited + } + return err +} + +// CheckService asks the object-storage service, with no binding, whether it +// reads a batch: a release that does is the one that also takes a per-call +// binding. One that does not — the composed agent predates it — is +// ErrBindingNotHonoured, and the host admits no object-storage source. +func CheckService(ctx context.Context, client storagev0.ObjectStorageClient) error { + ctx, cancel := context.WithTimeout(ctx, callTimeout) + defer cancel() + caps, err := client.Capabilities(ctx, &storagev0.CapabilitiesRequest{}) + if err != nil { + return storeError(err) + } + if caps.GetMaxGetManyItems() <= 0 { + return ErrBindingNotHonoured + } + return nil +} + +// Dial opens a connection to the object-storage service at addr, presenting +// token as x-codefly-token on every call. The service is reached in plaintext +// inside the cell, where the mesh secures the hop; an empty token dials +// without one, which is a deployed cell's profile. +func Dial(addr, token string) (*grpc.ClientConn, error) { + opts := []grpc.DialOption{grpc.WithTransportCredentials(insecure.NewCredentials())} + if token != "" { + opts = append(opts, grpc.WithPerRPCCredentials(serviceToken(token))) + } + return grpc.NewClient(addr, opts...) +} + +type serviceToken string + +func (t serviceToken) GetRequestMetadata(context.Context, ...string) (map[string]string, error) { + return map[string]string{"x-codefly-token": string(t)}, nil +} + +func (serviceToken) RequireTransportSecurity() bool { return false } diff --git a/module/services/accounts/code/pkg/datasource/objectstore/service_contract_test.go b/module/services/accounts/code/pkg/datasource/objectstore/service_contract_test.go new file mode 100644 index 000000000..d472a2b02 --- /dev/null +++ b/module/services/accounts/code/pkg/datasource/objectstore/service_contract_test.go @@ -0,0 +1,262 @@ +package objectstore + +import ( + "context" + "encoding/xml" + "errors" + "fmt" + "net" + "net/http" + "net/http/httptest" + "os" + "os/exec" + "path/filepath" + "sort" + "strings" + "sync" + "testing" + "time" + + storagev0 "github.com/codefly-dev/service-object-storage/gen/codefly/storage/v0" + + "accounts/pkg/datasource/connector" +) + +// The host's deletion authority rests on two things the object-storage service +// does, not the host: a bound listing's prefix is used literally, and a listing +// the store marks incomplete fails rather than answers as a complete, possibly +// empty, one. Both once lived only in the service's implementation, and both +// were wrong there. These tests hold the service the host actually pins to +// them: they build cmd/service-object-storage from the module go.mod requires, +// run it against a scripted S3 endpoint admitted as a composed dependency, and +// read through ServiceBucket. A service version that regressed either fails +// here, before a sync could tombstone a collection on it. + +// scriptedS3 answers ListObjectsV2 for one bucket, path-style, recording the +// prefix each request asked for. +type scriptedS3 struct { + mu sync.Mutex + keys map[string]string // key -> content + truncated bool // answer IsTruncated with no continuation token + prefixes []string +} + +type s3Contents struct { + Key string `xml:"Key"` + ETag string `xml:"ETag"` + Size int `xml:"Size"` + LastModified string `xml:"LastModified"` +} + +type s3ListResult struct { + XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ ListBucketResult"` + Name string `xml:"Name"` + Prefix string `xml:"Prefix"` + KeyCount int `xml:"KeyCount"` + MaxKeys int `xml:"MaxKeys"` + IsTruncated bool `xml:"IsTruncated"` + Contents []s3Contents `xml:"Contents"` +} + +func (s *scriptedS3) ServeHTTP(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet || r.URL.Query().Get("list-type") != "2" { + http.Error(w, "unscripted request "+r.Method+" "+r.URL.String(), http.StatusNotImplemented) + return + } + prefix := r.URL.Query().Get("prefix") + s.mu.Lock() + s.prefixes = append(s.prefixes, prefix) + res := s3ListResult{Name: "contract", Prefix: prefix, MaxKeys: 1000, IsTruncated: s.truncated} + var keys []string + for k := range s.keys { + if strings.HasPrefix(k, prefix) { + keys = append(keys, k) + } + } + sort.Strings(keys) + for _, k := range keys { + res.Contents = append(res.Contents, s3Contents{Key: k, ETag: etagOf(s.keys[k]), Size: len(s.keys[k]), LastModified: "2026-10-23T00:00:00.000Z"}) + } + s.mu.Unlock() + res.KeyCount = len(res.Contents) + w.Header().Set("Content-Type", "application/xml") + _, _ = w.Write([]byte(xml.Header)) + _ = xml.NewEncoder(w).Encode(res) +} + +func (s *scriptedS3) asked() []string { + s.mu.Lock() + defer s.mu.Unlock() + return append([]string(nil), s.prefixes...) +} + +var ( + gatewayOnce sync.Once + gatewayBin string + gatewayErr error +) + +// pinnedGateway builds the gateway binary of the service-object-storage module +// this go.mod pins, once per test binary. +func pinnedGateway(t *testing.T) string { + t.Helper() + gatewayOnce.Do(func() { + out, err := exec.Command("go", "list", "-m", "-f", "{{.Dir}}", "github.com/codefly-dev/service-object-storage").Output() + if err != nil { + gatewayErr = fmt.Errorf("locate the pinned service-object-storage module: %w", err) + return + } + dir := strings.TrimSpace(string(out)) + bin := filepath.Join(os.TempDir(), fmt.Sprintf("objectstore-contract-gateway-%d", os.Getpid())) + build := exec.Command("go", "build", "-o", bin, "./cmd/service-object-storage") + build.Dir = dir + build.Env = append(os.Environ(), "GOWORK=off", "GOFLAGS=-mod=mod", "CGO_ENABLED=0") + if out, err := build.CombinedOutput(); err != nil { + gatewayErr = fmt.Errorf("build the pinned gateway in %s: %w\n%s", dir, err, out) + return + } + gatewayBin = bin + }) + if gatewayErr != nil { + // Never a skip: a contract test that cannot run must not read as one + // that passed. + t.Fatal(gatewayErr) + } + return gatewayBin +} + +func freePort(t *testing.T) int { + t.Helper() + l, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer func() { _ = l.Close() }() + return l.Addr().(*net.TCPAddr).Port +} + +// startGateway runs the pinned gateway on its own in-memory bucket, admitting +// store — and nothing else non-public — as a composed dependency, and returns a +// client presenting its token. +func startGateway(t *testing.T, store *httptest.Server) storagev0.ObjectStorageClient { + t.Helper() + bin := pinnedGateway(t) + _, storePort, _ := net.SplitHostPort(strings.TrimPrefix(store.URL, "http://")) + listen := fmt.Sprintf("127.0.0.1:%d", freePort(t)) + cmd := exec.Command(bin) + cmd.Env = append(os.Environ(), + "SOS_LISTEN="+listen, "SOS_BACKEND=mem", "SOS_BUCKET=own", "SOS_AUTH_TOKEN=contract-token", + "SOS_COMPOSED_ENDPOINTS=localhost:"+storePort, + ) + var logs strings.Builder + cmd.Stdout, cmd.Stderr = &logs, &logs + if err := cmd.Start(); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { + _ = cmd.Process.Kill() + _ = cmd.Wait() + if t.Failed() { + t.Logf("gateway output:\n%s", logs.String()) + } + }) + deadline := time.Now().Add(20 * time.Second) + for { + c, err := net.DialTimeout("tcp", listen, 200*time.Millisecond) + if err == nil { + _ = c.Close() + break + } + if time.Now().After(deadline) { + t.Fatalf("the pinned gateway never listened on %s:\n%s", listen, logs.String()) + } + time.Sleep(100 * time.Millisecond) + } + cc, err := Dial(listen, "contract-token") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = cc.Close() }) + return storagev0.NewObjectStorageClient(cc) +} + +func contractBucket(client storagev0.ObjectStorageClient, store *httptest.Server, prefix string) *ServiceBucket { + _, port, _ := net.SplitHostPort(strings.TrimPrefix(store.URL, "http://")) + cfg := SourceConfig{Endpoint: "http://localhost:" + port, Region: "us-east-1", Bucket: "contract", Prefix: prefix} + return NewServiceBucket(client, cfg, "contract-key", "contract-secret") +} + +// TestPinnedServiceKeepsALiteralPrefix: the pinned service asks the store for +// exactly the source's prefix — a leading slash, a bare slash and a doubled +// separator are not normalized — and the host takes exactly the keys under it. +// A service that rewrote "/reports/" to "reports/" would list the wrong key +// space and let the closing listing tombstone the right one. +func TestPinnedServiceKeepsALiteralPrefix(t *testing.T) { + s3 := &scriptedS3{keys: map[string]string{ + "/reports/r.md": "slashed", "reports/r.md": "plain", "/x.md": "root", "a//b/c.md": "double", "a/b/c.md": "single", + }} + store := httptest.NewServer(s3) + defer store.Close() + client := startGateway(t, store) + for prefix, want := range map[string][]string{ + "/reports/": {"/reports/r.md"}, + "/": {"/reports/r.md", "/x.md"}, + "a//b/": {"a//b/c.md"}, + } { + items, err := contractBucket(client, store, prefix).List(context.Background()) + if err != nil { + t.Fatalf("prefix %q: %v", prefix, err) + } + var got []string + for _, it := range items { + got = append(got, it.Key) + } + sort.Strings(got) + asked := s3.asked() + if strings.Join(got, ",") != strings.Join(want, ",") || len(asked) == 0 || asked[len(asked)-1] != prefix { + t.Fatalf("prefix %q: listed %v after asking the store for %q, want %v listed under the literal prefix", prefix, got, asked, want) + } + } +} + +// TestPinnedServiceRefusesAnIncompleteListing: a store that marks its listing +// truncated without a continuation token has not listed the prefix whole. The +// pinned service must fail the call; answered as a complete — here empty — +// listing, the host's closing listing would tombstone every entry. +func TestPinnedServiceRefusesAnIncompleteListing(t *testing.T) { + for name, keys := range map[string]map[string]string{ + "empty": {}, + "nonempty": {"kb/a.md": "alpha"}, + } { + store := httptest.NewServer(&scriptedS3{keys: keys, truncated: true}) + client := startGateway(t, store) + items, err := contractBucket(client, store, "kb/").List(context.Background()) + store.Close() + if err == nil { + t.Fatalf("%s truncated listing with no continuation token = %d items and no error, want a refusal", name, len(items)) + } + if errors.Is(err, ErrBindingNotHonoured) { + t.Fatalf("%s: the pinned gateway does not honour the binding: %v", name, err) + } + } +} + +// TestPinnedServiceNamesATooLongListing: a store that answers a page with more +// entries than it was asked for has not answered the listing the host asked +// for. The pinned service refuses it and names the reason, and the host must +// read that reason as an incomplete listing — which fails the sync with +// nothing delivered — never as a throttle, which would block the credential +// for every source sharing it and retry the same listing forever. +func TestPinnedServiceNamesATooLongListing(t *testing.T) { + keys := map[string]string{} + for i := 0; i <= 1000; i++ { + keys[fmt.Sprintf("kb/%04d.md", i)] = "x" + } + store := httptest.NewServer(&scriptedS3{keys: keys}) + defer store.Close() + _, err := contractBucket(startGateway(t, store), store, "kb/").List(context.Background()) + var limited *connector.RateLimitedError + if !errors.Is(err, ErrListingIncomplete) || errors.As(err, &limited) { + t.Fatalf("a page of 1,001 entries = %v, want ErrListingIncomplete and no rate limit", err) + } +} diff --git a/module/services/accounts/code/pkg/datasource/objectstore/service_test.go b/module/services/accounts/code/pkg/datasource/objectstore/service_test.go new file mode 100644 index 000000000..91feebc11 --- /dev/null +++ b/module/services/accounts/code/pkg/datasource/objectstore/service_test.go @@ -0,0 +1,544 @@ +package objectstore + +import ( + "context" + "errors" + "fmt" + "io" + "net" + "sort" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + storagev0 "github.com/codefly-dev/service-object-storage/gen/codefly/storage/v0" + "google.golang.org/genproto/googleapis/rpc/errdetails" + "google.golang.org/grpc" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/credentials/insecure" + "google.golang.org/grpc/status" + "google.golang.org/grpc/test/bufconn" + "google.golang.org/protobuf/protoadapt" + "google.golang.org/protobuf/types/known/durationpb" + + "accounts/pkg/datasource/connector" +) + +// fakeService is the object-storage service's bound reads as its proto states +// them: keys relative to the binding's prefix, a listing paged by +// next_page_token, a GetMany at each item's If-Match that streams a header per +// item and the bytes of each one found at it. It checks the bound credential, +// can be throttled, counts every call, and records every binding it was handed. +type fakeService struct { + storagev0.UnimplementedObjectStorageServer + mu sync.Mutex + objects map[string]string // full key -> content + pageSize int + maxBatch int // max_get_many_items; 0 means 1000 + secret string // when set, the only secret the store accepts + err error // returned by every call when set + unbound bool // a release from before the binding: Capabilities reports no bound caps + unbatched bool // a release from before GetMany: no max_get_many_items + misorder bool // answers the first object under the second's index + stray string // a key List adds to its first page whatever the prefix + overfill bool // a List page one object past the page size it reports + stall time.Duration + truncate bool // sends one byte less of each object than its size + throttled atomic.Bool + authenticated atomic.Bool + calls atomic.Int64 + bindings []*storagev0.Binding + listReqs []*storagev0.ListRequest + getManys []*storagev0.GetManyRequest +} + +// call is what every RPC does first: count, record, then refuse as the +// service would — its own error, a refused credential, a throttle. +func (f *fakeService) call(b *storagev0.Binding) error { + f.calls.Add(1) + f.mu.Lock() + f.bindings = append(f.bindings, b) + err := f.err + f.mu.Unlock() + switch { + case err != nil: + return err + case f.secret != "" && b.GetCredential().GetAccessKey().GetSecret() != f.secret: + return status.Error(codes.PermissionDenied, "access denied") + case f.throttled.Load(): + return status.Error(codes.ResourceExhausted, "slow down") + } + f.authenticated.Store(true) + return nil +} + +func (f *fakeService) Capabilities(_ context.Context, req *storagev0.CapabilitiesRequest) (*storagev0.BackendCapabilities, error) { + if err := f.call(req.GetBinding()); err != nil { + return nil, err + } + if f.unbound { + return &storagev0.BackendCapabilities{Backend: "minio"}, nil + } + batch := f.maxBatch + switch { + case f.unbatched: + batch = 0 + case batch == 0: + batch = 1000 + } + return &storagev0.BackendCapabilities{Backend: "s3", MaxObjectBytes: MaxObjectBytes, MaxListLimit: int32(f.pageSize), MaxGetManyItems: int32(batch)}, nil +} + +func (f *fakeService) List(_ context.Context, req *storagev0.ListRequest) (*storagev0.ListResult, error) { + if err := f.call(req.GetBinding()); err != nil { + return nil, err + } + if f.stall > 0 { + time.Sleep(f.stall) + } + f.mu.Lock() + defer f.mu.Unlock() + f.listReqs = append(f.listReqs, req) + folder := req.GetBinding().GetPrefix() + var rel []string + for key := range f.objects { + if r, ok := strings.CutPrefix(key, folder); ok && strings.HasPrefix(r, req.GetPrefix()) { + rel = append(rel, r) + } + } + sort.Strings(rel) + start := 0 + if req.GetPageToken() != "" { + _, _ = fmt.Sscan(req.GetPageToken(), &start) + } + size := f.pageSize + if f.overfill { + size++ + } + end := min(start+size, len(rel)) + res := &storagev0.ListResult{} + if f.stray != "" && start == 0 { + res.Objects = append(res.Objects, &storagev0.ObjectInfo{Key: f.stray, Etag: etagOf("stray"), Size: 5}) + } + for _, r := range rel[start:end] { + res.Objects = append(res.Objects, &storagev0.ObjectInfo{Key: r, Etag: etagOf(f.objects[folder+r]), Size: int64(len(f.objects[folder+r]))}) + } + if end < len(rel) { + res.NextPageToken = fmt.Sprint(end) + } + return res, nil +} + +func (f *fakeService) GetMany(req *storagev0.GetManyRequest, stream grpc.ServerStreamingServer[storagev0.GetManyResponse]) error { + if err := f.call(req.GetBinding()); err != nil { + return err + } + f.mu.Lock() + f.getManys = append(f.getManys, req) + contents := make([]*string, len(req.GetItems())) + for i, item := range req.GetItems() { + if content, ok := f.objects[req.GetBinding().GetPrefix()+item.GetKey()]; ok { + contents[i] = &content + } + } + f.mu.Unlock() + for i, item := range req.GetItems() { + header := &storagev0.GetManyHeader{Index: int32(i)} + if f.misorder && i == 0 { + header.Index = 1 + } + switch content := contents[i]; { + case content == nil: + header.Outcome = storagev0.GetManyOutcome_GET_MANY_OUTCOME_NOT_FOUND + case item.GetIfMatch() != "" && item.GetIfMatch() != etagOf(*content): + header.Outcome = storagev0.GetManyOutcome_GET_MANY_OUTCOME_CHANGED + case len(*content) > MaxObjectBytes: + header.Outcome = storagev0.GetManyOutcome_GET_MANY_OUTCOME_TOO_LARGE + default: + header.Outcome = storagev0.GetManyOutcome_GET_MANY_OUTCOME_OK + header.Info = &storagev0.ObjectInfo{Key: item.GetKey(), Etag: etagOf(*content), Size: int64(len(*content)), ContentType: "text/markdown"} + } + if err := stream.Send(&storagev0.GetManyResponse{Kind: &storagev0.GetManyResponse_Header{Header: header}}); err != nil { + return err + } + if header.Outcome != storagev0.GetManyOutcome_GET_MANY_OUTCOME_OK { + continue + } + body := []byte(*contents[i]) + if f.truncate { + body = body[:len(body)-1] + } + for chunk := range slicesChunk(body, 3) { + if err := stream.Send(&storagev0.GetManyResponse{Kind: &storagev0.GetManyResponse_Data{Data: chunk}}); err != nil { + return err + } + } + } + return nil +} + +func slicesChunk(b []byte, n int) func(func([]byte) bool) { + return func(yield func([]byte) bool) { + for len(b) > 0 { + k := min(n, len(b)) + if !yield(b[:k]) { + return + } + b = b[k:] + } + } +} + +func serve(t *testing.T, f *fakeService) storagev0.ObjectStorageClient { + t.Helper() + lis := bufconn.Listen(1 << 20) + srv := grpc.NewServer() + storagev0.RegisterObjectStorageServer(srv, f) + go func() { _ = srv.Serve(lis) }() + t.Cleanup(srv.Stop) + cc, err := grpc.NewClient("passthrough:///bufnet", + grpc.WithContextDialer(func(ctx context.Context, _ string) (net.Conn, error) { return lis.DialContext(ctx) }), + grpc.WithTransportCredentials(insecure.NewCredentials())) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = cc.Close() }) + return storagev0.NewObjectStorageClient(cc) +} + +var serviceConfig = SourceConfig{Endpoint: "https://store.example.com", Region: "eu-west-1", Bucket: "acme-docs", Prefix: "kb/2026-"} + +// TestServiceBucketListsTheSourcesKeyPrefixThroughTheBinding: a source's +// prefix is a literal key prefix. The binding carries none — the service would +// normalize it as a folder — and every page filters on the prefix as written, +// so every item comes back under its full key, the identity the host's +// deliveries carry. +func TestServiceBucketListsTheSourcesKeyPrefixThroughTheBinding(t *testing.T) { + f := &fakeService{pageSize: 2, objects: map[string]string{ + "kb/2026-a.md": "alpha", "kb/2026-b.md": "bravo", "kb/2026-c.md": "charlie", "kb/2025-z.md": "old", "other/2026-x.md": "elsewhere", + }} + b := NewServiceBucket(serve(t, f), serviceConfig, "AKIAEXAMPLE", "sentinel-secret") + items, err := b.List(context.Background()) + if err != nil { + t.Fatal(err) + } + var keys []string + for _, it := range items { + keys = append(keys, it.Key) + if it.ETag != etagOf(f.objects[it.Key]) || it.Size != int64(len(f.objects[it.Key])) { + t.Fatalf("item %+v does not carry its ETag and size", it) + } + } + if want := []string{"kb/2026-a.md", "kb/2026-b.md", "kb/2026-c.md"}; strings.Join(keys, ",") != strings.Join(want, ",") { + t.Fatalf("listed %v, want %v", keys, want) + } + if len(f.listReqs) != 2 || f.listReqs[0].GetPrefix() != "kb/2026-" || f.listReqs[1].GetPageToken() == "" { + t.Fatalf("list requests = %v, want two pages filtered on the literal prefix", f.listReqs) + } + bound := f.bindings[0] + if bound.GetKind() != "s3" || bound.GetEndpoint() != serviceConfig.Endpoint || bound.GetRegion() != "eu-west-1" || + bound.GetBucket() != "acme-docs" || bound.GetPrefix() != "" || + bound.GetCredential().GetAccessKey().GetId() != "AKIAEXAMPLE" || bound.GetCredential().GetAccessKey().GetSecret() != "sentinel-secret" { + t.Fatalf("binding = %v, want the source's store and credential and no prefix", bound) + } +} + +// TestServiceBucketKeepsALiteralPrefixLiteral: "/reports/" is not "reports/", +// and "/" is not the whole bucket. A prefix the service would rewrite as a +// folder reaches it as written, so a source never lists another key space and +// then reconciles deletions against it. +func TestServiceBucketKeepsALiteralPrefixLiteral(t *testing.T) { + objects := map[string]string{"/reports/a.md": "slashed", "reports/b.md": "plain", "//double.md": "double"} + for prefix, want := range map[string]string{"/reports/": "/reports/a.md", "reports/": "reports/b.md", "//": "//double.md"} { + f := &fakeService{pageSize: 10, objects: objects} + cfg := serviceConfig + cfg.Prefix = prefix + b := NewServiceBucket(serve(t, f), cfg, "id", "secret") + items, err := b.List(context.Background()) + if err != nil || len(items) != 1 || items[0].Key != want { + t.Fatalf("prefix %q listed %v (%v), want only %s", prefix, items, err, want) + } + if err := b.Read(context.Background(), items, func(Item, string, io.Reader) error { return nil }); err != nil { + t.Fatalf("prefix %q: reading %s = %v", prefix, want, err) + } + if got := f.getManys[0].GetItems()[0].GetKey(); got != want { + t.Fatalf("prefix %q: read key %q, want the full key %q", prefix, got, want) + } + } + f := &fakeService{pageSize: 10, objects: objects} + cfg := serviceConfig + cfg.Prefix = "/" + items, err := NewServiceBucket(serve(t, f), cfg, "id", "secret").List(context.Background()) + if err != nil || len(items) != 2 { + t.Fatalf(`prefix "/" listed %v (%v), want only the two keys that start with "/"`, items, err) + } +} + +// TestServiceBucketRefusesAListingOutsideItsScope: a key the literal prefix +// does not name, or a page past the service's own page size, refuses the whole +// listing — nothing from it reaches a version the closing listing would +// reconcile deletions against. +func TestServiceBucketRefusesAListingOutsideItsScope(t *testing.T) { + objects := map[string]string{"kb/2026-a.md": "alpha", "kb/2026-b.md": "bravo", "kb/2026-c.md": "charlie"} + for name, f := range map[string]*fakeService{ + "a key outside the prefix": {pageSize: 10, stray: "other/x.md", objects: objects}, + "a page past its page size": {pageSize: 2, overfill: true, objects: objects}, + } { + if items, err := NewServiceBucket(serve(t, f), serviceConfig, "id", "secret").List(context.Background()); !errors.Is(err, errOutsideScope) { + t.Fatalf("%s: List = %v, %v, want errOutsideScope", name, items, err) + } + } +} + +// TestServiceBucketBoundsAStalledCall: a store that answers and then stalls +// does not hold the sync past the call's deadline. +func TestServiceBucketBoundsAStalledCall(t *testing.T) { + defer func(d time.Duration) { callTimeout = d }(callTimeout) + callTimeout = 50 * time.Millisecond + f := &fakeService{pageSize: 10, stall: time.Second, objects: map[string]string{"kb/2026-a.md": "alpha"}} + start := time.Now() + _, err := NewServiceBucket(serve(t, f), serviceConfig, "id", "secret").List(context.Background()) + if status.Code(err) != codes.DeadlineExceeded || time.Since(start) > 500*time.Millisecond { + t.Fatalf("a stalled List = %v after %s, want DeadlineExceeded at the call's deadline", err, time.Since(start)) + } +} + +func TestServiceBucketRefusesAListingPastItsPageBound(t *testing.T) { + f := &fakeService{pageSize: 1, objects: map[string]string{}} + for i := 0; i <= maxListPages; i++ { + f.objects[fmt.Sprintf("kb/2026-%02d.md", i)] = "x" + } + _, err := NewServiceBucket(serve(t, f), serviceConfig, "id", "secret").List(context.Background()) + if !errors.Is(err, ErrListingIncomplete) { + t.Fatalf("a listing of %d pages = %v, want ErrListingIncomplete", maxListPages+1, err) + } +} + +// TestServiceBucketReadsABatchInOneCall: a batch is one GetMany of each key +// relative to the binding, at If-Match its listed ETag, answered in request +// order; a batch past the service's max_get_many_items is split; an object +// rewritten or deleted since its listing is ErrObjectChanged, and one over the +// service's cap is the envelope's ErrItemTooLarge. +func TestServiceBucketReadsABatchInOneCall(t *testing.T) { + f := &fakeService{pageSize: 10, objects: map[string]string{"kb/2026-a.md": "alpha", "kb/2026-b.md": "bravo", "kb/2026-c.md": "charlie"}} + b := NewServiceBucket(serve(t, f), serviceConfig, "id", "secret") + items, err := b.List(context.Background()) + if err != nil { + t.Fatal(err) + } + read := func(b *ServiceBucket, items []Item) (map[string]string, error) { + got := map[string]string{} + err := b.Read(context.Background(), items, func(it Item, contentType string, r io.Reader) error { + body, _ := io.ReadAll(r) + got[it.Key] = contentType + ":" + string(body) + return nil + }) + return got, err + } + got, err := read(b, items) + if err != nil { + t.Fatal(err) + } + if got["kb/2026-a.md"] != "text/markdown:alpha" || got["kb/2026-b.md"] != "text/markdown:bravo" || got["kb/2026-c.md"] != "text/markdown:charlie" { + t.Fatalf("read %v", got) + } + if len(f.getManys) != 1 || len(f.getManys[0].GetItems()) != 3 || + f.getManys[0].GetItems()[0].GetKey() != "kb/2026-a.md" || f.getManys[0].GetItems()[0].GetIfMatch() != etagOf("alpha") { + t.Fatalf("GetMany calls = %v, want one for the batch, full keys, at If-Match", f.getManys) + } + + small := &fakeService{pageSize: 10, maxBatch: 2, objects: f.objects} + if got, err := read(NewServiceBucket(serve(t, small), serviceConfig, "id", "secret"), items); err != nil || len(got) != 3 || len(small.getManys) != 2 { + t.Fatalf("a batch past max_get_many_items: read %d objects in %d calls (%v), want 3 in 2", len(got), len(small.getManys), err) + } + + f.objects["kb/2026-a.md"] = "alpha, rewritten" + delete(f.objects, "kb/2026-b.md") + for _, it := range items[:2] { + if _, err := read(b, []Item{it}); !errors.Is(err, ErrObjectChanged) { + t.Fatalf("reading %s after it changed = %v, want ErrObjectChanged", it.Key, err) + } + } + // An object the service refuses as over its cap is skipped and the rest of + // the batch is still read; the read then names what it skipped. + f.objects["kb/2026-c.md"] = strings.Repeat("x", MaxObjectBytes+1) + f.objects["kb/2026-d.md"] = "delta" + mixed := []Item{ + {Key: "kb/2026-c.md", ETag: etagOf(f.objects["kb/2026-c.md"]), Size: MaxObjectBytes + 1}, + {Key: "kb/2026-d.md", ETag: etagOf("delta"), Size: 5}, + } + got, err = read(b, mixed) + var skipped *TooLargeError + if !errors.As(err, &skipped) || !errors.Is(err, connector.ErrItemTooLarge) || len(skipped.Keys) != 1 || skipped.Keys[0] != "kb/2026-c.md" { + t.Fatalf("reading past an object over the service's cap = %v, want the oversized one named and skipped", err) + } + if got["kb/2026-d.md"] != "text/markdown:delta" { + t.Fatalf("read %v: the object after the oversized one was not read", got) + } +} + +// TestServiceBucketRefusesAnAnswerItCannotAttribute: an object answered out of +// request order, or whose bytes stop short of its listed size, is refused — +// never delivered as another object's content or as a truncated file. +func TestServiceBucketRefusesAnAnswerItCannotAttribute(t *testing.T) { + objects := map[string]string{"kb/2026-a.md": "alpha", "kb/2026-b.md": "bravo"} + items := []Item{{Key: "kb/2026-a.md", ETag: etagOf("alpha"), Size: 5}, {Key: "kb/2026-b.md", ETag: etagOf("bravo"), Size: 5}} + for name, f := range map[string]*fakeService{ + "out of order": {pageSize: 10, misorder: true, objects: objects}, + "short": {pageSize: 10, truncate: true, objects: objects}, + } { + delivered := 0 + err := NewServiceBucket(serve(t, f), serviceConfig, "id", "secret").Read(context.Background(), items, func(Item, string, io.Reader) error { + delivered++ + return nil + }) + if err == nil || delivered != 0 { + t.Fatalf("%s answer: Read = %v after delivering %d objects, want a refusal before any", name, err, delivered) + } + } +} + +// TestServiceBucketRefusesAServiceThatIgnoresTheBinding: a service from before +// the binding drops the field and answers for its own bucket. Nothing it would +// list or read is taken for the source's. +func TestServiceBucketRefusesAServiceThatIgnoresTheBinding(t *testing.T) { + f := &fakeService{pageSize: 10, unbound: true, objects: map[string]string{"kb/2026-a.md": "alpha"}} + b := NewServiceBucket(serve(t, f), serviceConfig, "id", "secret") + if _, err := b.List(context.Background()); !errors.Is(err, ErrBindingNotHonoured) { + t.Fatalf("List on an unbinding service = %v, want ErrBindingNotHonoured", err) + } + err := b.Read(context.Background(), []Item{{Key: "kb/2026-a.md", ETag: etagOf("alpha"), Size: 5}}, func(Item, string, io.Reader) error { return nil }) + if !errors.Is(err, ErrBindingNotHonoured) || len(f.listReqs) != 0 || len(f.getManys) != 0 { + t.Fatalf("Read = %v after %d lists and %d reads; an unbinding service is never listed or read", err, len(f.listReqs), len(f.getManys)) + } + unbatched := &fakeService{pageSize: 10, unbatched: true, objects: f.objects} + if _, err := NewServiceBucket(serve(t, unbatched), serviceConfig, "id", "secret").List(context.Background()); !errors.Is(err, ErrBindingNotHonoured) { + t.Fatalf("List on a service with no batched read = %v, want ErrBindingNotHonoured", err) + } +} + +// TestCheckServiceAdmitsOnlyABatchedRead: the host's start-up question to the +// composed service. One that reports no batch size predates GetMany and the +// binding, and the host wires no object storage. +func TestCheckServiceAdmitsOnlyABatchedRead(t *testing.T) { + if err := CheckService(context.Background(), serve(t, &fakeService{pageSize: 10})); err != nil { + t.Fatalf("a service with a batched read = %v, want admitted", err) + } + if err := CheckService(context.Background(), serve(t, &fakeService{pageSize: 10, unbatched: true})); !errors.Is(err, ErrBindingNotHonoured) { + t.Fatalf("a service with no batched read = %v, want ErrBindingNotHonoured", err) + } +} + +// TestServiceBucketTypesTheServicesRefusals: a refused credential or endpoint +// is the envelope's credential refusal, a throttle is a rate limit carrying the +// service's retry delay, and anything else keeps its status. +func TestServiceBucketTypesTheServicesRefusals(t *testing.T) { + throttle, _ := status.New(codes.ResourceExhausted, "slow down").WithDetails(&errdetails.RetryInfo{RetryDelay: durationpb.New(30 * time.Second)}) + reasoned := func(c codes.Code, reason string, details ...protoadapt.MessageV1) error { + st, _ := status.New(c, strings.ToLower(reason)).WithDetails(append([]protoadapt.MessageV1{&errdetails.ErrorInfo{Domain: errorDomain, Reason: reason}}, details...)...) + return st.Err() + } + notThrottle := func(err error) bool { + var limited *connector.RateLimitedError + return !errors.As(err, &limited) + } + for _, tc := range []struct { + name string + err error + check func(error) bool + }{ + {"permission denied", status.Error(codes.PermissionDenied, `binding endpoint "10.0.0.1": destination is not a public address`), func(err error) bool { + return errors.Is(err, connector.ErrCredentialRefused) + }}, + {"throttled", throttle.Err(), func(err error) bool { + var limited *connector.RateLimitedError + return errors.As(err, &limited) && limited.ResetAt.After(time.Now().Add(20*time.Second)) + }}, + // The reason, not the code, decides (service-object-storage v0.0.19). + {"guard refused the endpoint", reasoned(codes.PermissionDenied, "DESTINATION_REFUSED"), func(err error) bool { + return errors.Is(err, ErrDestinationRefused) && !errors.Is(err, connector.ErrCredentialRefused) + }}, + {"incomplete listing", reasoned(codes.Internal, "LISTING_INCOMPLETE"), func(err error) bool { + return errors.Is(err, ErrListingIncomplete) && notThrottle(err) + }}, + {"store rejected the credential", reasoned(codes.PermissionDenied, "CREDENTIAL_REJECTED"), func(err error) bool { + return errors.Is(err, connector.ErrCredentialRefused) + }}, + {"listing too long", reasoned(codes.ResourceExhausted, "LISTING_TOO_LONG"), func(err error) bool { + return errors.Is(err, ErrListingIncomplete) && notThrottle(err) + }}, + {"response too large", reasoned(codes.ResourceExhausted, "RESPONSE_TOO_LARGE"), func(err error) bool { + return errors.Is(err, ErrListingIncomplete) && notThrottle(err) + }}, + {"object too large", reasoned(codes.ResourceExhausted, "OBJECT_TOO_LARGE"), func(err error) bool { + return errors.Is(err, connector.ErrItemTooLarge) && notThrottle(err) + }}, + {"store throttled", reasoned(codes.ResourceExhausted, "STORE_THROTTLED", &errdetails.RetryInfo{RetryDelay: durationpb.New(30 * time.Second)}), func(err error) bool { + var limited *connector.RateLimitedError + return errors.As(err, &limited) && limited.ResetAt.After(time.Now().Add(20*time.Second)) + }}, + // A missing bucket is the store's error, never an empty listing: an + // empty one would be a version that names no object, and the sync's + // closing listing would tombstone the whole collection against it. + {"missing bucket", status.Error(codes.NotFound, "no such bucket"), func(err error) bool { + return status.Code(err) == codes.NotFound + }}, + {"invalid binding", status.Error(codes.InvalidArgument, "bad bucket"), func(err error) bool { + return status.Code(err) == codes.InvalidArgument && !errors.Is(err, connector.ErrCredentialRefused) + }}, + } { + t.Run(tc.name, func(t *testing.T) { + f := &fakeService{pageSize: 10, err: tc.err} + if _, err := NewServiceBucket(serve(t, f), serviceConfig, "id", "secret").List(context.Background()); !tc.check(err) { + t.Fatalf("List under %v = %v", tc.err, err) + } + }) + } +} + +// TestFilesConnectorOverTheService runs the connector on the service-backed +// Bucket: a snapshot, a fetch at its version, and a deletion seen by the next +// change set — what a sync does, through the service's own wire. +func TestFilesConnectorOverTheService(t *testing.T) { + f := &fakeService{pageSize: 1, objects: map[string]string{"kb/2026-a.md": "alpha", "kb/2026-b.md": "bravo", "kb/2026-draft.md": "wip"}} + client := serve(t, f) + cfg := serviceConfig + cfg.Scope = Scope{InScope: func(key string) bool { return !strings.HasSuffix(key, "draft.md") }} + conn := NewFilesConnector(func(context.Context, connector.Source) (Bucket, error) { + return NewServiceBucket(client, cfg, "id", "secret"), nil + }, &memoryManifests{}) + src := connector.Source{ID: "aaaaaaaa-0000-0000-0000-000000000001", OrgID: "11111111-1111-1111-1111-111111111111", BoundaryNodeID: "bbbbbbbb-0000-0000-0000-000000000001", Config: cfg} + ctx := context.Background() + first, err := conn.Changes(ctx, src, "") + if err != nil { + t.Fatal(err) + } + if len(first.Changes) != 2 { + t.Fatalf("snapshot = %+v, want the two objects the scope admits", first.Changes) + } + var refs []connector.FileRef + for _, ch := range first.Changes { + refs = append(refs, connector.FileRef{ItemID: ch.Locator, ItemVersion: ch.ItemVersion}) + } + fetched := map[string]string{} + if err := conn.FetchFiles(ctx, src, first.To, refs, func(file connector.File, r io.Reader) error { + body, _ := io.ReadAll(r) + fetched[file.Locator] = string(body) + return nil + }); err != nil { + t.Fatal(err) + } + if fetched["kb/2026-a.md"] != "alpha" || fetched["kb/2026-b.md"] != "bravo" { + t.Fatalf("fetched %v", fetched) + } + delete(f.objects, "kb/2026-b.md") + next, err := conn.Changes(ctx, src, first.To) + if err != nil { + t.Fatal(err) + } + if len(next.Changes) != 1 || next.Changes[0].Kind != connector.ChangeDeleted || next.Changes[0].Locator != "kb/2026-b.md" { + t.Fatalf("after a delete, changes = %+v, want kb/2026-b.md deleted", next.Changes) + } +} diff --git a/module/services/accounts/code/pkg/gen/saas/accounts/v1/datasource.pb.go b/module/services/accounts/code/pkg/gen/saas/accounts/v1/datasource.pb.go index 8bae49fb6..2f590aefa 100644 --- a/module/services/accounts/code/pkg/gen/saas/accounts/v1/datasource.pb.go +++ b/module/services/accounts/code/pkg/gen/saas/accounts/v1/datasource.pb.go @@ -1249,9 +1249,25 @@ type UploadDatasourceConfig struct { // AWS-style access key id. The matching secret access key is the credential. AccessKeyId string `protobuf:"bytes,5,opt,name=access_key_id,json=accessKeyId,proto3" json:"access_key_id,omitempty"` // Upper bound on objects fetched per sync. Zero applies the connector default. - MaxObjects uint32 `protobuf:"varint,6,opt,name=max_objects,json=maxObjects,proto3" json:"max_objects,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache + // Only objects that pass the filters below count against it. + MaxObjects uint32 `protobuf:"varint,6,opt,name=max_objects,json=maxObjects,proto3" json:"max_objects,omitempty"` + // Which keys under the prefix are ingested. Each pattern is an RE2 regular + // expression matched against the object key with the prefix removed, + // unanchored and case-sensitive (anchor with ^ and $, fold case with (?i)). + // An empty include admits every key; an exclude match always wins. Applied + // when the prefix is listed, before anything is fetched, so an excluded + // object is never read and never counts against max_objects. An invalid + // expression is refused when the source is added or updated, never at sync: + // the call fails INVALID_ARGUMENT with a google.rpc.BadRequest violation, and + // a message that starts with the field's path, e.g. + // "upload.include_patterns[1]: …". + IncludePatterns []string `protobuf:"bytes,7,rep,name=include_patterns,json=includePatterns,proto3" json:"include_patterns,omitempty"` + ExcludePatterns []string `protobuf:"bytes,8,rep,name=exclude_patterns,json=excludePatterns,proto3" json:"exclude_patterns,omitempty"` + // Case-insensitive key suffixes, e.g. [".md", ".pdf"], intersected with the + // patterns. Empty admits every type. GitHub's file_extensions semantics. + FileExtensions []string `protobuf:"bytes,9,rep,name=file_extensions,json=fileExtensions,proto3" json:"file_extensions,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache } func (x *UploadDatasourceConfig) Reset() { @@ -1326,6 +1342,27 @@ func (x *UploadDatasourceConfig) GetMaxObjects() uint32 { return 0 } +func (x *UploadDatasourceConfig) GetIncludePatterns() []string { + if x != nil { + return x.IncludePatterns + } + return nil +} + +func (x *UploadDatasourceConfig) GetExcludePatterns() []string { + if x != nil { + return x.ExcludePatterns + } + return nil +} + +func (x *UploadDatasourceConfig) GetFileExtensions() []string { + if x != nil { + return x.FileExtensions + } + return nil +} + // Datasource is one connected external datasource. It is org-scoped and carries // only non-secret projection fields; credential material is deliberately absent. // Exactly one provider config (github, api, crawler, or upload) is set, matching @@ -2084,9 +2121,14 @@ type UpdateSourceRequest struct { // on every successful sync, which would refuse a perfectly good edit for the // sole reason that the source had reconciled since the form was opened. What // an editor needs guarded is what they were looking at. - Expected *SourceFetchScope `protobuf:"bytes,6,opt,name=expected,proto3" json:"expected,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache + Expected *SourceFetchScope `protobuf:"bytes,6,opt,name=expected,proto3" json:"expected,omitempty"` + // An object-storage source's key patterns (see UploadDatasourceConfig), + // replaced whole like every field here. Its file_extensions is field 5; its + // branch and paths must be empty. Refused for a github source. + IncludePatterns []string `protobuf:"bytes,7,rep,name=include_patterns,json=includePatterns,proto3" json:"include_patterns,omitempty"` + ExcludePatterns []string `protobuf:"bytes,8,rep,name=exclude_patterns,json=excludePatterns,proto3" json:"exclude_patterns,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache } func (x *UpdateSourceRequest) Reset() { @@ -2161,6 +2203,20 @@ func (x *UpdateSourceRequest) GetExpected() *SourceFetchScope { return nil } +func (x *UpdateSourceRequest) GetIncludePatterns() []string { + if x != nil { + return x.IncludePatterns + } + return nil +} + +func (x *UpdateSourceRequest) GetExcludePatterns() []string { + if x != nil { + return x.ExcludePatterns + } + return nil +} + // SourceFetchScope is what a source fetches — the three values UpdateSource // replaces — carried on its own so a request can state the scope it expects to // be replacing as well as the one it wants. @@ -2171,12 +2227,14 @@ func (x *UpdateSourceRequest) GetExpected() *SourceFetchScope { // edit at all. Size is still bounded, because the comparison reads the whole // thing. type SourceFetchScope struct { - state protoimpl.MessageState `protogen:"open.v1"` - Branch string `protobuf:"bytes,1,opt,name=branch,proto3" json:"branch,omitempty"` - Paths []string `protobuf:"bytes,2,rep,name=paths,proto3" json:"paths,omitempty"` - FileExtensions []string `protobuf:"bytes,3,rep,name=file_extensions,json=fileExtensions,proto3" json:"file_extensions,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache + state protoimpl.MessageState `protogen:"open.v1"` + Branch string `protobuf:"bytes,1,opt,name=branch,proto3" json:"branch,omitempty"` + Paths []string `protobuf:"bytes,2,rep,name=paths,proto3" json:"paths,omitempty"` + FileExtensions []string `protobuf:"bytes,3,rep,name=file_extensions,json=fileExtensions,proto3" json:"file_extensions,omitempty"` + IncludePatterns []string `protobuf:"bytes,4,rep,name=include_patterns,json=includePatterns,proto3" json:"include_patterns,omitempty"` + ExcludePatterns []string `protobuf:"bytes,5,rep,name=exclude_patterns,json=excludePatterns,proto3" json:"exclude_patterns,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache } func (x *SourceFetchScope) Reset() { @@ -2230,6 +2288,20 @@ func (x *SourceFetchScope) GetFileExtensions() []string { return nil } +func (x *SourceFetchScope) GetIncludePatterns() []string { + if x != nil { + return x.IncludePatterns + } + return nil +} + +func (x *SourceFetchScope) GetExcludePatterns() []string { + if x != nil { + return x.ExcludePatterns + } + return nil +} + type UpdateSourceResponse struct { state protoimpl.MessageState `protogen:"open.v1"` Datasource *Datasource `protobuf:"bytes,1,opt,name=datasource,proto3" json:"datasource,omitempty"` @@ -7474,7 +7546,7 @@ const file_saas_accounts_v1_datasource_proto_rawDesc = "" + "\x17CrawlerDatasourceConfig\x12\x1f\n" + "\vsitemap_url\x18\x01 \x01(\tR\n" + "sitemapUrl\x12\x1b\n" + - "\tmax_pages\x18\x02 \x01(\rR\bmaxPages\"\xc1\x01\n" + + "\tmax_pages\x18\x02 \x01(\rR\bmaxPages\"\x98\x03\n" + "\x16UploadDatasourceConfig\x12\x1a\n" + "\bendpoint\x18\x01 \x01(\tR\bendpoint\x12\x16\n" + "\x06region\x18\x02 \x01(\tR\x06region\x12\x16\n" + @@ -7482,7 +7554,10 @@ const file_saas_accounts_v1_datasource_proto_rawDesc = "" + "\x06prefix\x18\x04 \x01(\tR\x06prefix\x12\"\n" + "\raccess_key_id\x18\x05 \x01(\tR\vaccessKeyId\x12\x1f\n" + "\vmax_objects\x18\x06 \x01(\rR\n" + - "maxObjects\"\x83\t\n" + + "maxObjects\x12<\n" + + "\x10include_patterns\x18\a \x03(\tB\x11\xbaH\x0e\x92\x01\v\x10\x10\"\ar\x05\x10\x01\x18\x80\x02R\x0fincludePatterns\x12<\n" + + "\x10exclude_patterns\x18\b \x03(\tB\x11\xbaH\x0e\x92\x01\v\x10\x10\"\ar\x05\x10\x01\x18\x80\x02R\x0fexcludePatterns\x12Y\n" + + "\x0ffile_extensions\x18\t \x03(\tB0\xbaH-\x92\x01*\x10 \"&r$\x10\x02\x18!2\x1e^\\.[A-Za-z0-9][A-Za-z0-9._-]*$R\x0efileExtensions\"\x83\t\n" + "\n" + "Datasource\x12\x0e\n" + "\x02id\x18\x01 \x01(\tR\x02id\x12\x15\n" + @@ -7550,19 +7625,23 @@ const file_saas_accounts_v1_datasource_proto_rawDesc = "" + "\x11AddSourceResponse\x12<\n" + "\n" + "datasource\x18\x01 \x01(\v2\x1c.saas.accounts.v1.DatasourceR\n" + - "datasource\"\xbe\x02\n" + + "datasource\"\xba\x03\n" + "\x13UpdateSourceRequest\x12\x1f\n" + "\x06org_id\x18\x01 \x01(\tB\b\xbaH\x05r\x03\xb0\x01\x01R\x05orgId\x12\x18\n" + "\x02id\x18\x02 \x01(\tB\b\xbaH\x05r\x03\xb0\x01\x01R\x02id\x12 \n" + "\x06branch\x18\x03 \x01(\tB\b\xbaH\x05r\x03\x18\xff\x01R\x06branch\x12'\n" + "\x05paths\x18\x04 \x03(\tB\x11\xbaH\x0e\x92\x01\v\x10@\"\ar\x05\x10\x01\x18\x80\x04R\x05paths\x12Y\n" + "\x0ffile_extensions\x18\x05 \x03(\tB0\xbaH-\x92\x01*\x10 \"&r$\x10\x02\x18!2\x1e^\\.[A-Za-z0-9][A-Za-z0-9._-]*$R\x0efileExtensions\x12F\n" + - "\bexpected\x18\x06 \x01(\v2\".saas.accounts.v1.SourceFetchScopeB\x06\xbaH\x03\xc8\x01\x01R\bexpected\"\x98\x01\n" + + "\bexpected\x18\x06 \x01(\v2\".saas.accounts.v1.SourceFetchScopeB\x06\xbaH\x03\xc8\x01\x01R\bexpected\x12<\n" + + "\x10include_patterns\x18\a \x03(\tB\x11\xbaH\x0e\x92\x01\v\x10\x10\"\ar\x05\x10\x01\x18\x80\x02R\x0fincludePatterns\x12<\n" + + "\x10exclude_patterns\x18\b \x03(\tB\x11\xbaH\x0e\x92\x01\v\x10\x10\"\ar\x05\x10\x01\x18\x80\x02R\x0fexcludePatterns\"\x90\x02\n" + "\x10SourceFetchScope\x12 \n" + "\x06branch\x18\x01 \x01(\tB\b\xbaH\x05r\x03\x18\xff\x01R\x06branch\x12'\n" + "\x05paths\x18\x02 \x03(\tB\x11\xbaH\x0e\x92\x01\v\x10@\"\ar\x05\x10\x01\x18\x80\x04R\x05paths\x129\n" + "\x0ffile_extensions\x18\x03 \x03(\tB\x10\xbaH\r\x92\x01\n" + - "\x10 \"\x06r\x04\x10\x01\x18!R\x0efileExtensions\"k\n" + + "\x10 \"\x06r\x04\x10\x01\x18!R\x0efileExtensions\x12:\n" + + "\x10include_patterns\x18\x04 \x03(\tB\x0f\xbaH\f\x92\x01\t\x10\x10\"\x05r\x03\x18\x80\x02R\x0fincludePatterns\x12:\n" + + "\x10exclude_patterns\x18\x05 \x03(\tB\x0f\xbaH\f\x92\x01\t\x10\x10\"\x05r\x03\x18\x80\x02R\x0fexcludePatterns\"k\n" + "\x14UpdateSourceResponse\x12<\n" + "\n" + "datasource\x18\x01 \x01(\v2\x1c.saas.accounts.v1.DatasourceR\n" + diff --git a/module/services/accounts/code/pkg/infra/postgres_datasource_sources.go b/module/services/accounts/code/pkg/infra/postgres_datasource_sources.go index b08e1458c..53a45ffb0 100644 --- a/module/services/accounts/code/pkg/infra/postgres_datasource_sources.go +++ b/module/services/accounts/code/pkg/infra/postgres_datasource_sources.go @@ -637,6 +637,32 @@ func (s *PostgresStore) SetDatasourceSourceSynced(ctx context.Context, orgID, id return err } +// RecordDatasourceSnapshotSent stamps a sync that sent its closing snapshot: +// the sync time and the number of objects the snapshot named, which is the +// host's record of whether the source has files (migration 60). +func (s *PostgresStore) RecordDatasourceSnapshotSent(ctx context.Context, orgID, id string, syncedAt time.Time, items int) error { + _, err := s.getQueryExecutor(ctx).Exec(ctx, ` + UPDATE datasource_sources + SET last_synced_at = $3, last_snapshot_items = $4, updated_at = NOW() + WHERE org_id = $1 AND id = $2`, orgID, id, syncedAt, items) + return err +} + +// DatasourceSourceLastSnapshot reads how many objects the source's last +// closing snapshot named; known is false when none was ever recorded. +func (s *PostgresStore) DatasourceSourceLastSnapshot(ctx context.Context, orgID, id string) (int, bool, error) { + var items *int + err := s.getQueryExecutor(ctx).QueryRow(ctx, ` + SELECT last_snapshot_items FROM datasource_sources WHERE org_id = $1 AND id = $2`, orgID, id).Scan(&items) + if errors.Is(err, pgx.ErrNoRows) { + return 0, false, business.ErrDatasourceSourceNotFound + } + if err != nil || items == nil { + return 0, false, err + } + return *items, true, nil +} + // LockDatasourceSourceCredentialRef reads the credential envelope under a row // lock so a refresh-and-rotate cycle (OAuth 2.0) serializes against a concurrent // sync of the same source: the second caller blocks here until the first commits, diff --git a/module/services/accounts/code/pkg/infra/postgres_object_manifests.go b/module/services/accounts/code/pkg/infra/postgres_object_manifests.go new file mode 100644 index 000000000..6d3abdbf8 --- /dev/null +++ b/module/services/accounts/code/pkg/infra/postgres_object_manifests.go @@ -0,0 +1,58 @@ +package infra + +import ( + "context" + "encoding/json" + "errors" + + "github.com/jackc/pgx/v5" + + "accounts/pkg/datasource/objectstore" +) + +// retainedObjectManifests is how many listings a source keeps: the one its +// cursor names, the one a sync in flight just listed, and slack for a sync that +// listed and then failed before the cursor moved. A version past it resyncs. +const retainedObjectManifests = 4 + +// PutDatasourceObjectManifest retains one listing of an object-storage source +// under its version, inside the source's org transaction, and prunes all but +// the newest retained listings. A version is the listing's own digest, so a +// second put of it only marks it newest. +func (s *PostgresStore) PutDatasourceObjectManifest(ctx context.Context, org, source, version string, objects []objectstore.Item) error { + if objects == nil { + objects = []objectstore.Item{} + } + encoded, err := json.Marshal(objects) + if err != nil { + return err + } + q := s.getQueryExecutor(ctx) + if _, err := q.Exec(ctx, `INSERT INTO datasource_object_manifests (org_id,source_id,version,objects,retained_at) VALUES ($1,$2,$3,$4,clock_timestamp()) + ON CONFLICT (org_id,source_id,version) DO UPDATE SET retained_at = clock_timestamp()`, org, source, version, encoded); err != nil { + return err + } + _, err = q.Exec(ctx, `DELETE FROM datasource_object_manifests WHERE org_id=$1 AND source_id=$2 AND version NOT IN ( + SELECT version FROM datasource_object_manifests WHERE org_id=$1 AND source_id=$2 ORDER BY retained_at DESC, version LIMIT $3)`, + org, source, retainedObjectManifests) + return err +} + +// GetDatasourceObjectManifest returns the listing a source retained under +// version, and false when it never retained one or has pruned it. +func (s *PostgresStore) GetDatasourceObjectManifest(ctx context.Context, org, source, version string) ([]objectstore.Item, bool, error) { + var encoded []byte + err := s.getQueryExecutor(ctx).QueryRow(ctx, `SELECT objects FROM datasource_object_manifests WHERE org_id=$1 AND source_id=$2 AND version=$3`, + org, source, version).Scan(&encoded) + if errors.Is(err, pgx.ErrNoRows) { + return nil, false, nil + } + if err != nil { + return nil, false, err + } + var objects []objectstore.Item + if err := json.Unmarshal(encoded, &objects); err != nil { + return nil, false, err + } + return objects, true, nil +} diff --git a/module/services/accounts/code/pkg/infra/postgres_object_manifests_test.go b/module/services/accounts/code/pkg/infra/postgres_object_manifests_test.go new file mode 100644 index 000000000..f854bea5b --- /dev/null +++ b/module/services/accounts/code/pkg/infra/postgres_object_manifests_test.go @@ -0,0 +1,135 @@ +//go:build !pure + +package infra_test + +import ( + "context" + "fmt" + "strings" + "testing" + "time" + + "accounts/pkg/datasource/objectstore" + + "github.com/stretchr/testify/require" +) + +func manifestVersion(i int) string { + return fmt.Sprintf("%016x.%064x", 0xabc, i) +} + +// TestPostgresDatasourceObjectManifestsRetainTheNewest pins the retained +// listings an object-storage source diffs against: a listing round-trips under +// its version, a source keeps only its newest few, a re-put of a version makes +// it newest again, and another organization reads none of them. +func TestPostgresDatasourceObjectManifestsRetainTheNewest(t *testing.T) { + userID := seedUser(t) + orgID := seedOrg(t, userID) + sourceID := seedDatasourceSource(t, orgID) + listing := []objectstore.Item{{Key: "kb/a.md", ETag: `"e1"`, Size: 5}, {Key: "kb/b.md", ETag: `"e2"`, Size: 7}} + + put := func(i int) { + require.NoError(t, testStore.WithOrgTx(testCtx, orgID, func(ctx context.Context) error { + return testStore.PutDatasourceObjectManifest(ctx, orgID, sourceID, manifestVersion(i), listing) + })) + } + get := func(org string, i int) ([]objectstore.Item, bool) { + var items []objectstore.Item + var ok bool + require.NoError(t, testStore.WithOrgTx(testCtx, org, func(ctx context.Context) error { + var err error + items, ok, err = testStore.GetDatasourceObjectManifest(ctx, orgID, sourceID, manifestVersion(i)) + return err + })) + return items, ok + } + + put(0) + items, ok := get(orgID, 0) + require.True(t, ok) + require.Equal(t, listing, items) + + for i := 1; i <= 3; i++ { + put(i) + } + put(0) // re-listed: newest again, so version 1 is now the oldest + put(4) + _, ok = get(orgID, 1) + require.False(t, ok, "the oldest listing past the retained count is pruned") + for _, i := range []int{0, 2, 3, 4} { + _, ok = get(orgID, i) + require.True(t, ok, "listing %d is among the newest and must be retained", i) + } + + otherOrg := seedOrg(t, seedUser(t)) + _, ok = get(otherOrg, 4) + require.False(t, ok, "another organization must not read a source's listings") +} + +// TestPostgresDatasourceObjectManifestRefusesAMalformedVersion: the version is +// the connector's digest, never a free string. +func TestPostgresDatasourceObjectManifestRefusesAMalformedVersion(t *testing.T) { + userID := seedUser(t) + orgID := seedOrg(t, userID) + sourceID := seedDatasourceSource(t, orgID) + err := testStore.WithOrgTx(testCtx, orgID, func(ctx context.Context) error { + return testStore.PutDatasourceObjectManifest(ctx, orgID, sourceID, "not-a-version", nil) + }) + require.Error(t, err) + require.True(t, strings.Contains(err.Error(), "check"), "want the version CHECK to refuse it, got %v", err) +} + +// TestPostgresDatasourceSnapshotCountRoundTrips pins the host's record of +// whether an object-storage source has files, which the empty-listing +// invariant reads: unknown until a closing snapshot is recorded, then the count +// that snapshot named — zero included — and the sync time with it; another +// organization reads nothing of it. +func TestPostgresDatasourceSnapshotCountRoundTrips(t *testing.T) { + userID := seedUser(t) + orgID := seedOrg(t, userID) + sourceID := seedDatasourceSource(t, orgID) + last := func(org string) (int, bool, error) { + var items int + var known bool + err := testStore.WithOrgTx(testCtx, org, func(ctx context.Context) error { + var err error + items, known, err = testStore.DatasourceSourceLastSnapshot(ctx, orgID, sourceID) + return err + }) + return items, known, err + } + record := func(items int) { + require.NoError(t, testStore.WithOrgTx(testCtx, orgID, func(ctx context.Context) error { + return testStore.RecordDatasourceSnapshotSent(ctx, orgID, sourceID, time.Now().UTC(), items) + })) + } + + _, known, err := last(orgID) + require.NoError(t, err) + require.False(t, known, "a source that never sent a snapshot has no recorded count") + + record(3) + items, known, err := last(orgID) + require.NoError(t, err) + require.True(t, known) + require.Equal(t, 3, items) + var synced *time.Time + require.NoError(t, testStore.WithOrgTx(testCtx, orgID, func(ctx context.Context) error { + source, err := testStore.GetDatasourceSource(ctx, orgID, sourceID) + if err == nil && source != nil { + synced = source.LastSyncedAt + } + return err + })) + require.NotNil(t, synced, "recording a snapshot records the sync time") + + record(0) + items, known, err = last(orgID) + require.NoError(t, err) + require.True(t, known) + require.Equal(t, 0, items, "an empty snapshot is recorded as zero, not as unknown") + + otherOrg := seedOrg(t, seedUser(t)) + _, known, err = last(otherOrg) + require.False(t, known && err == nil, "another organization must not read a source's snapshot count") +} diff --git a/module/services/accounts/code/pkg/infra/postgres_role_hardening_test.go b/module/services/accounts/code/pkg/infra/postgres_role_hardening_test.go index 75045d864..e1dc94424 100644 --- a/module/services/accounts/code/pkg/infra/postgres_role_hardening_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_role_hardening_test.go @@ -94,6 +94,7 @@ var appTenantRelationPrivileges = map[string]relationPrivileges{ "installations": {selectRows: true, insertRows: true, updateRows: true, deleteRows: true}, "datasource_sources": {selectRows: true, insertRows: true, updateRows: true, deleteRows: true}, "datasource_source_operations": {selectRows: true, insertRows: true, updateRows: true, deleteRows: true}, + "datasource_object_manifests": {selectRows: true, insertRows: true, updateRows: true, deleteRows: true}, "datasource_operation_attempts": {selectRows: true, insertRows: true, deleteRows: true}, "codefly_effect_receipts": {selectRows: true, insertRows: true, deleteRows: true}, "datasource_oauth_consumed_states": {selectRows: true, insertRows: true, deleteRows: true}, @@ -380,11 +381,12 @@ func TestControlPlaneRelationGrantsAreExact(t *testing.T) { if relation == "executable_artifact_approvals" { want = relationPrivileges{} } - // Callable-source declarations, OAuth state, attempts and receipts - // are tenant-only. The control plane must not read or mutate them. + // Callable-source declarations, OAuth state, attempts, receipts and + // retained object listings are tenant-only. The control plane must not read or mutate them. switch relation { case "datasource_source_operations", "datasource_oauth_consumed_states", - "datasource_operation_attempts", "codefly_effect_receipts": + "datasource_operation_attempts", "codefly_effect_receipts", + "datasource_object_manifests": want = relationPrivileges{} } if relation == "feature_flags" { diff --git a/module/services/accounts/code/pkg/infra/postgres_webhooks_test.go b/module/services/accounts/code/pkg/infra/postgres_webhooks_test.go index c3568d915..fa4e41375 100644 --- a/module/services/accounts/code/pkg/infra/postgres_webhooks_test.go +++ b/module/services/accounts/code/pkg/infra/postgres_webhooks_test.go @@ -102,7 +102,7 @@ func runPostgresInfraTests(m *testing.M) int { setupDone := testdb.MeasureSetup("infra-db", []string{"store"}, 90*time.Second) deps, err := sdk.WithDependencies(ctx, sdk.WithDebug(), - sdk.WithExcludedDependencies("cache", "vault"), + sdk.WithExcludedDependencies("object-storage", "cache", "vault"), sdk.WithNamingScope("test-infra"), sdk.WithTimeout(90*time.Second), sdk.WithSilence("store"), diff --git a/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt b/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt index 46ea6b557..c25965cea 100644 --- a/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt +++ b/module/services/accounts/code/pkg/infra/testdata/control_plane_call_sites.txt @@ -39,7 +39,7 @@ pkg/business/datasource_deliveries.go: (*Service).RunDatasourceReconcile: WithCo pkg/business/datasource_deliveries.go: (*Service).RunDatasourceReconcile: WithControlPlane pkg/business/datasource_deliveries.go: (*Service).advanceCursor: WithControlPlane pkg/business/datasource_deliveries.go: (*Service).allocateOrdinal: WithControlPlane -pkg/business/datasource_deliveries.go: (*Service).parkUnreadablePublicSource: WithControlPlane +pkg/business/datasource_deliveries.go: (*Service).parkDatasourceSource: WithControlPlane pkg/business/datasource_deliveries.go: (*Service).snapshotAt: WithControlPlane pkg/business/datasource_deliveries.go: (*Service).snapshotAt: WithControlPlane pkg/business/datasource_github_app_push.go: (*Service).FanOutGitHubAppPush: WithControlPlane diff --git a/module/services/accounts/code/pkg/redisintegration/main_test.go b/module/services/accounts/code/pkg/redisintegration/main_test.go index 10bbfa6d3..4bc73f2cc 100644 --- a/module/services/accounts/code/pkg/redisintegration/main_test.go +++ b/module/services/accounts/code/pkg/redisintegration/main_test.go @@ -46,7 +46,7 @@ func runRedisIntegrationTests(m *testing.M) int { setupDone := testdb.MeasureSetup("redis-db", []string{"cache", "store"}, 120*time.Second) deps, err := sdk.WithDependencies(ctx, sdk.WithDebug(), - sdk.WithExcludedDependencies("vault"), + sdk.WithExcludedDependencies("object-storage", "vault"), sdk.WithNamingScope("redis-integration"), sdk.WithTimeout(120*time.Second), sdk.WithSilence("store", "cache"), diff --git a/module/services/accounts/code/pkg/relationcatalog/catalog.go b/module/services/accounts/code/pkg/relationcatalog/catalog.go index 421275ec3..f7a859592 100644 --- a/module/services/accounts/code/pkg/relationcatalog/catalog.go +++ b/module/services/accounts/code/pkg/relationcatalog/catalog.go @@ -151,6 +151,7 @@ var authorities = map[string]Authority{ "dashboards": {Scope: ScopeTenant, PolicyShape: ShapeDirect, ScopeColumn: "org_id"}, "datasource_sources": {Scope: ScopeTenant, PolicyShape: ShapeDirect, ScopeColumn: "org_id"}, "datasource_source_operations": {Scope: ScopeTenant, PolicyShape: ShapeDirect, ScopeColumn: "org_id"}, + "datasource_object_manifests": {Scope: ScopeTenant, PolicyShape: ShapeDirect, ScopeColumn: "org_id"}, "datasource_operation_attempts": {Scope: ScopeTenant, PolicyShape: ShapeDirect, ScopeColumn: "org_id"}, "codefly_effect_receipts": {Scope: ScopeTenant, PolicyShape: ShapeDirect, ScopeColumn: "tenant"}, "datasource_oauth_consumed_states": {Scope: ScopeTenant, PolicyShape: ShapeDirect, ScopeColumn: "org_id"}, diff --git a/module/services/accounts/code/work.go b/module/services/accounts/code/work.go index 42c023bfc..57714e12b 100644 --- a/module/services/accounts/code/work.go +++ b/module/services/accounts/code/work.go @@ -17,6 +17,7 @@ import ( "accounts/pkg/business" "accounts/pkg/datasource" "accounts/pkg/datasource/apisource" + "accounts/pkg/datasource/objectstore" "accounts/pkg/email" "accounts/pkg/eventcatalog" "accounts/pkg/githubconnector" @@ -45,9 +46,11 @@ import ( "github.com/codefly-dev/core/workloadidentity/native" "github.com/codefly-dev/module-saas-starter/module/frontendpayload" codefly "github.com/codefly-dev/sdk-go" + storagev0 "github.com/codefly-dev/service-object-storage/gen/codefly/storage/v0" "github.com/google/uuid" "go.opentelemetry.io/otel" otelmetric "go.opentelemetry.io/otel/metric" + "google.golang.org/grpc" ) func init() { @@ -356,6 +359,19 @@ func doWork(ctx context.Context) (clean Clean, err error) { // documents module consumes. GITHUB_API_BASE_URL overrides api.github.com for // GitHub Enterprise or tests. service.SetDatasourceConnector(cipher, jobStore, os.Getenv("GITHUB_API_BASE_URL")) + // Object-storage sources are read through the composed object-storage + // service, under a per-call binding of each source's bucket and credential. + objectStorage, err := dialObjectStorage(ctx) + if err != nil { + return nil, err + } + if objectStorage != nil { + if err := service.SetObjectStorage(ctx, storagev0.NewObjectStorageClient(objectStorage)); err != nil { + // Not a start failure: the host runs, and object storage is refused + // by name in the catalog, at AddSource and at sync. + wool.Get(ctx).Warn("the composed object-storage service cannot read a tenant's bucket; object-storage datasources are unavailable", wool.ErrField(err)) + } + } // Every provider credential's operations are metered in one window shared by // every replica, so a sync a person starts is served before background work. service.SetDatasourceBudgetStore(store) @@ -1417,6 +1433,9 @@ func doWork(ctx context.Context) (clean Clean, err error) { sw.Info("redis cache closed") } } + if objectStorage != nil { + _ = objectStorage.Close() + } sw.Info("closing store") store.Close() sw.Info("store closed") @@ -3007,3 +3026,24 @@ func registerMembershipCacheHealth(c *membership.Cache) { return nil }, gauge) } + +// dialObjectStorage opens the connection to the composed object-storage +// service. Codefly resolves its endpoint; the service's agent hands its per-run +// token to consumers as the "token" secret of its object-storage configuration. +// A deployed cell publishes no such token — it runs the service anonymous and +// gates callers at the mesh — so an absent token dials without one. A +// composition that leaves the service out has no endpoint for it: the host +// then runs without object storage, and an object-storage sync refuses by name. +func dialObjectStorage(ctx context.Context) (*grpc.ClientConn, error) { + network := codefly.For(ctx).Service("object-storage").Endpoint("grpc").API("grpc").NetworkInstance() + if network == nil || network.Address == "" { + wool.Get(ctx).Info("no object-storage service is composed; object-storage datasources cannot sync") + return nil, nil + } + token, _ := codefly.For(ctx).Service("object-storage").Secret("object-storage", "token") + conn, err := objectstore.Dial(network.Address, token) + if err != nil { + return nil, fmt.Errorf("dial object-storage: %w", err) + } + return conn, nil +} diff --git a/module/services/accounts/proto/saas/accounts/v1/datasource.proto b/module/services/accounts/proto/saas/accounts/v1/datasource.proto index 60efdaffa..c95696bd1 100644 --- a/module/services/accounts/proto/saas/accounts/v1/datasource.proto +++ b/module/services/accounts/proto/saas/accounts/v1/datasource.proto @@ -189,7 +189,48 @@ message UploadDatasourceConfig { // AWS-style access key id. The matching secret access key is the credential. string access_key_id = 5; // Upper bound on objects fetched per sync. Zero applies the connector default. + // Only objects that pass the filters below count against it. uint32 max_objects = 6; + // Which keys under the prefix are ingested. Each pattern is an RE2 regular + // expression matched against the object key with the prefix removed, + // unanchored and case-sensitive (anchor with ^ and $, fold case with (?i)). + // An empty include admits every key; an exclude match always wins. Applied + // when the prefix is listed, before anything is fetched, so an excluded + // object is never read and never counts against max_objects. An invalid + // expression is refused when the source is added or updated, never at sync: + // the call fails INVALID_ARGUMENT with a google.rpc.BadRequest violation, and + // a message that starts with the field's path, e.g. + // "upload.include_patterns[1]: …". + repeated string include_patterns = 7 [(buf.validate.field).repeated = { + max_items: 16 + items: { + string: { + min_len: 1 + max_len: 256 + } + } + }]; + repeated string exclude_patterns = 8 [(buf.validate.field).repeated = { + max_items: 16 + items: { + string: { + min_len: 1 + max_len: 256 + } + } + }]; + // Case-insensitive key suffixes, e.g. [".md", ".pdf"], intersected with the + // patterns. Empty admits every type. GitHub's file_extensions semantics. + repeated string file_extensions = 9 [(buf.validate.field).repeated = { + max_items: 32 + items: { + string: { + min_len: 2 + max_len: 33 + pattern: "^\\.[A-Za-z0-9][A-Za-z0-9._-]*$" + } + } + }]; } // Datasource is one connected external datasource. It is org-scoped and carries @@ -437,6 +478,27 @@ message UpdateSourceRequest { // sole reason that the source had reconciled since the form was opened. What // an editor needs guarded is what they were looking at. SourceFetchScope expected = 6 [(buf.validate.field).required = true]; + // An object-storage source's key patterns (see UploadDatasourceConfig), + // replaced whole like every field here. Its file_extensions is field 5; its + // branch and paths must be empty. Refused for a github source. + repeated string include_patterns = 7 [(buf.validate.field).repeated = { + max_items: 16 + items: { + string: { + min_len: 1 + max_len: 256 + } + } + }]; + repeated string exclude_patterns = 8 [(buf.validate.field).repeated = { + max_items: 16 + items: { + string: { + min_len: 1 + max_len: 256 + } + } + }]; } // SourceFetchScope is what a source fetches — the three values UpdateSource @@ -467,6 +529,17 @@ message SourceFetchScope { max_len: 33 } } + }]; repeated string include_patterns = 4 [(buf.validate.field).repeated = { + max_items: 16 + items: { + string: {max_len: 256} + } + }]; + repeated string exclude_patterns = 5 [(buf.validate.field).repeated = { + max_items: 16 + items: { + string: {max_len: 256} + } }]; } diff --git a/module/services/accounts/service.codefly.yaml b/module/services/accounts/service.codefly.yaml index ef8e12610..b8376764f 100644 --- a/module/services/accounts/service.codefly.yaml +++ b/module/services/accounts/service.codefly.yaml @@ -19,6 +19,11 @@ service-dependencies: endpoints: - name: read - name: write + # Object-storage sources are read through this service, under a per-call + # binding of the source's bucket and credential (pkg/datasource/objectstore). + - name: object-storage + endpoints: + - name: grpc - name: store endpoints: - name: tcp diff --git a/module/services/frontend/code/package-lock.json b/module/services/frontend/code/package-lock.json index 03f7edaef..f843f8fbc 100644 --- a/module/services/frontend/code/package-lock.json +++ b/module/services/frontend/code/package-lock.json @@ -15,7 +15,7 @@ "@bufbuild/protobuf": "^2.11.0", "@codefly-dev/saas-plugin-contract": "2.4.4", "@codefly-dev/saas-plugin-react": "0.4.5", - "@codefly-dev/saas-sdk": "0.3.37", + "@codefly-dev/saas-sdk": "0.3.40", "@codefly-dev/saas-ui": "0.13.0", "@codefly-dev/ui": "0.13.0", "@codefly/saas-plugin-manifest": "0.1.2", @@ -15627,7 +15627,7 @@ }, "packages/codefly-ui": { "name": "@codefly-dev/ui", - "version": "0.13.2", + "version": "0.13.3", "license": "Apache-2.0", "dependencies": { "@base-ui/react": "^1.3.0", @@ -15742,7 +15742,7 @@ }, "packages/saas-sdk": { "name": "@codefly-dev/saas-sdk", - "version": "0.3.39", + "version": "0.3.40", "license": "Apache-2.0", "dependencies": { "@bufbuild/protobuf": "^2.11.0", @@ -15776,7 +15776,7 @@ }, "packages/saas-ui": { "name": "@codefly-dev/saas-ui", - "version": "0.13.2", + "version": "0.13.3", "license": "Apache-2.0", "dependencies": { "@opentelemetry/api": "1.9.1", @@ -15787,7 +15787,7 @@ }, "devDependencies": { "@bufbuild/protobuf": "^2.11.0", - "@codefly-dev/saas-sdk": "^0.3.35", + "@codefly-dev/saas-sdk": "^0.3.40", "@codefly-dev/ui": "^0.13.0", "@connectrpc/connect": "^2.1.1", "@connectrpc/connect-web": "^2.1.1", @@ -15804,7 +15804,7 @@ }, "peerDependencies": { "@bufbuild/protobuf": ">=2", - "@codefly-dev/saas-sdk": "^0.3.35", + "@codefly-dev/saas-sdk": "^0.3.40", "@codefly-dev/ui": "^0.13.0", "@connectrpc/connect": ">=2", "@connectrpc/connect-web": ">=2", diff --git a/module/services/frontend/code/package.json b/module/services/frontend/code/package.json index 75d2a636f..4dd0ffcb3 100644 --- a/module/services/frontend/code/package.json +++ b/module/services/frontend/code/package.json @@ -39,7 +39,7 @@ "@bufbuild/protobuf": "^2.11.0", "@codefly-dev/saas-plugin-contract": "2.4.4", "@codefly-dev/saas-plugin-react": "0.4.5", - "@codefly-dev/saas-sdk": "0.3.37", + "@codefly-dev/saas-sdk": "0.3.40", "@codefly-dev/saas-ui": "0.13.0", "@codefly-dev/ui": "0.13.0", "@codefly/saas-plugin-manifest": "0.1.2", diff --git a/module/services/frontend/code/packages/codefly-ui/package.json b/module/services/frontend/code/packages/codefly-ui/package.json index 30b7404dd..f3b2f3e2a 100644 --- a/module/services/frontend/code/packages/codefly-ui/package.json +++ b/module/services/frontend/code/packages/codefly-ui/package.json @@ -1,6 +1,6 @@ { "name": "@codefly-dev/ui", - "version": "0.13.2", + "version": "0.13.3", "description": "The shared, versioned Codefly SaaS frontend kit: the plugin host and the tokens-as-data skin mechanism, standalone so the host and Module-Federation remotes import and dedupe one copy.", "repository": { "type": "git", diff --git a/module/services/frontend/code/packages/codefly-ui/src/__tests__/no-raw-type-primitives.test.ts b/module/services/frontend/code/packages/codefly-ui/src/__tests__/no-raw-type-primitives.test.ts index 5d4c3d36b..790d6254b 100644 --- a/module/services/frontend/code/packages/codefly-ui/src/__tests__/no-raw-type-primitives.test.ts +++ b/module/services/frontend/code/packages/codefly-ui/src/__tests__/no-raw-type-primitives.test.ts @@ -78,7 +78,7 @@ const SAAS_UI_BASELINE: Record = { // Lowered from 16 when the credential block moved into // `datasources/credential-mode.tsx`, which carries none: the shared copy // names its slots, so the debt left behind shrank rather than moving. - "datasources/connect-github-form.tsx": 11, + "datasources/connect-github-form.tsx": 10, "datasources/datasources-panel.tsx": 15, }; const saasUiDir = resolve(srcDir, "../../saas-ui/src"); diff --git a/module/services/frontend/code/packages/saas-sdk/generated/contract/catalog.codefly.json b/module/services/frontend/code/packages/saas-sdk/generated/contract/catalog.codefly.json index 890632088..68c3d388e 100644 --- a/module/services/frontend/code/packages/saas-sdk/generated/contract/catalog.codefly.json +++ b/module/services/frontend/code/packages/saas-sdk/generated/contract/catalog.codefly.json @@ -10,7 +10,7 @@ "kind": "protobuf", "package": "", "path": "contract.binpb", - "digest": "sha256:03c0df4fc804a9bcc85b5c436e696717323c2da4ac0a9dd9f9c3a6bcc602fc8b", + "digest": "sha256:da3be80f4ed222b07d24e2b23d5eb797a36b1334fef560b24aeecf2850ac48d6", "services": [ { "name": "AccessibleScopeService", diff --git a/module/services/frontend/code/packages/saas-sdk/generated/contract/contract.binpb b/module/services/frontend/code/packages/saas-sdk/generated/contract/contract.binpb index 74969bd57..4d1a16547 100644 Binary files a/module/services/frontend/code/packages/saas-sdk/generated/contract/contract.binpb and b/module/services/frontend/code/packages/saas-sdk/generated/contract/contract.binpb differ diff --git a/module/services/frontend/code/packages/saas-sdk/generated/library.codefly.yaml b/module/services/frontend/code/packages/saas-sdk/generated/library.codefly.yaml index 14ebda8a9..5cb12c298 100644 --- a/module/services/frontend/code/packages/saas-sdk/generated/library.codefly.yaml +++ b/module/services/frontend/code/packages/saas-sdk/generated/library.codefly.yaml @@ -12,7 +12,7 @@ sources: module: saas-starter service: accounts endpoint: connect - contract-digest: sha256:10a231ea430313364937808160138bca599ac012230ad70b7047eac27764836a + contract-digest: sha256:743b63bad9a546f105e994c754d845865e189465e6c72816e5785efe38ab7831 services: - AccessibleScopeService - AuditService diff --git a/module/services/frontend/code/packages/saas-sdk/generated/typescript/src/gen/saas/accounts/v1/datasource_pb.ts b/module/services/frontend/code/packages/saas-sdk/generated/typescript/src/gen/saas/accounts/v1/datasource_pb.ts index efec36dae..224ac5185 100644 --- a/module/services/frontend/code/packages/saas-sdk/generated/typescript/src/gen/saas/accounts/v1/datasource_pb.ts +++ b/module/services/frontend/code/packages/saas-sdk/generated/typescript/src/gen/saas/accounts/v1/datasource_pb.ts @@ -18,7 +18,7 @@ import type { JsonObject, Message } from "@bufbuild/protobuf"; * Describes the file saas/accounts/v1/datasource.proto. */ export const file_saas_accounts_v1_datasource: GenFile = /*@__PURE__*/ - fileDesc("CiFzYWFzL2FjY291bnRzL3YxL2RhdGFzb3VyY2UucHJvdG8SEHNhYXMuYWNjb3VudHMudjEiXgoWR2l0SHViRGF0YXNvdXJjZUNvbmZpZxIMCgRyZXBvGAEgASgJEg0KBXBhdGhzGAIgAygJEg4KBmJyYW5jaBgDIAEoCRIXCg9maWxlX2V4dGVuc2lvbnMYBCADKAki9AEKD0FwaU9BdXRoMkNvbmZpZxI2CgVncmFudBgEIAEoDjInLnNhYXMuYWNjb3VudHMudjEuQXBpT0F1dGgyQ29uZmlnLkdyYW50EhEKCXRva2VuX3VybBgBIAEoCRIRCgljbGllbnRfaWQYAiABKAkSDgoGc2NvcGVzGAMgAygJInMKBUdyYW50EhUKEUdSQU5UX1VOU1BFQ0lGSUVEEAASFwoTR1JBTlRfUkVGUkVTSF9UT0tFThABEhwKGEdSQU5UX0NMSUVOVF9DUkVERU5USUFMUxACEhwKGEdSQU5UX0FVVEhPUklaQVRJT05fQ09ERRADIuoBChNBcGlEYXRhc291cmNlQ29uZmlnEhAKCGJhc2VfdXJsGAEgASgJEhUKDXJlc291cmNlX3BhdGgYAiABKAkSPAoPY3JlZGVudGlhbF9raW5kGAMgASgOMiMuc2Fhcy5hY2NvdW50cy52MS5BcGlDcmVkZW50aWFsS2luZBIZChFjcmVkZW50aWFsX2hlYWRlchgEIAEoCRIeChZjcmVkZW50aWFsX3F1ZXJ5X3BhcmFtGAUgASgJEjEKBm9hdXRoMhgGIAEoCzIhLnNhYXMuYWNjb3VudHMudjEuQXBpT0F1dGgyQ29uZmlnIkEKF0NyYXdsZXJEYXRhc291cmNlQ29uZmlnEhMKC3NpdGVtYXBfdXJsGAEgASgJEhEKCW1heF9wYWdlcxgCIAEoDSKGAQoWVXBsb2FkRGF0YXNvdXJjZUNvbmZpZxIQCghlbmRwb2ludBgBIAEoCRIOCgZyZWdpb24YAiABKAkSDgoGYnVja2V0GAMgASgJEg4KBnByZWZpeBgEIAEoCRIVCg1hY2Nlc3Nfa2V5X2lkGAUgASgJEhMKC21heF9vYmplY3RzGAYgASgNIoMHCgpEYXRhc291cmNlEgoKAmlkGAEgASgJEg4KBm9yZ19pZBgCIAEoCRI2Cghwcm92aWRlchgDIAEoDjIkLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZVByb3ZpZGVyEjgKBmdpdGh1YhgFIAEoCzIoLnNhYXMuYWNjb3VudHMudjEuR2l0SHViRGF0YXNvdXJjZUNvbmZpZxIyCgZzdGF0dXMYBiABKA4yIi5zYWFzLmFjY291bnRzLnYxLkRhdGFzb3VyY2VTdGF0dXMSGgoSd2ViaG9va19jb25maWd1cmVkGAcgASgIEi4KCmNyZWF0ZWRfYXQYCCABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEi4KCnVwZGF0ZWRfYXQYCSABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEjIKDmxhc3Rfc3luY2VkX2F0GAogASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIyCgNhcGkYCyABKAsyJS5zYWFzLmFjY291bnRzLnYxLkFwaURhdGFzb3VyY2VDb25maWcSOgoHY3Jhd2xlchgMIAEoCzIpLnNhYXMuYWNjb3VudHMudjEuQ3Jhd2xlckRhdGFzb3VyY2VDb25maWcSOAoGdXBsb2FkGA0gASgLMiguc2Fhcy5hY2NvdW50cy52MS5VcGxvYWREYXRhc291cmNlQ29uZmlnEhgKEGJvdW5kYXJ5X25vZGVfaWQYDiABKAkSNAoQbGFzdF9pbmdlc3RlZF9hdBgPIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASHAoUbGFzdF9pbmdlc3RlZF9jb21taXQYECABKAkSFQoNc3RhdHVzX3JlYXNvbhgRIAEoCRIWCg5ib3VuZGFyeV9sYWJlbBgSIAEoCRISCgpjb25mb3JtYW50GBMgASgIEhcKD2NvbmZvcm1hbmNlX2dhcBgUIAEoCRI/Cg1saXZlX2RlbGl2ZXJ5GBUgASgOMiguc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlTGl2ZURlbGl2ZXJ5EjUKEnJlY29uY2lsZV9pbnRlcnZhbBgWIAEoCzIZLmdvb2dsZS5wcm90b2J1Zi5EdXJhdGlvbkoECAQQBVIRdGFyZ2V0X2NvbGxlY3Rpb24isgMKFkFkZEdpdEh1YlNvdXJjZVJlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARI7CgRyZXBvGAIgASgJQi26SCpyKBADGP8BMiFeW0EtWmEtejAtOV8uLV0rL1tBLVphLXowLTlfLi1dKyQSIAoFcGF0aHMYAyADKAlCEbpIDpIBCxBAIgdyBRABGIAEEhgKBmJyYW5jaBgEIAEoCUIIukgFcgMY/wESJAoQYm91bmRhcnlfbm9kZV9pZBgIIAEoCUIIukgFcgOwAQFIABImChBjb2xsZWN0aW9uX2xhYmVsGAkgASgJQgq6SAdyBRABGP8BSAASHgoMYWNjZXNzX3Rva2VuGAYgASgJQgi6SAVyAxiACBIgCg53ZWJob29rX3NlY3JldBgHIAEoCUIIukgFcgMYgAgSSQoPZmlsZV9leHRlbnNpb25zGAogAygJQjC6SC2SASoQICImciQQAhghMh5eXC5bQS1aYS16MC05XVtBLVphLXowLTkuXy1dKiRCEQoIYm91bmRhcnkSBbpIAggBSgQIBRAGUhF0YXJnZXRfY29sbGVjdGlvbiJLChdBZGRHaXRIdWJTb3VyY2VSZXNwb25zZRIwCgpkYXRhc291cmNlGAEgASgLMhwuc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlIsgEChBBZGRTb3VyY2VSZXF1ZXN0EhgKBm9yZ19pZBgBIAEoCUIIukgFcgOwAQESQgoIcHJvdmlkZXIYAiABKA4yJC5zYWFzLmFjY291bnRzLnYxLkRhdGFzb3VyY2VQcm92aWRlckIKukgHggEEEAEgABI6CgZnaXRodWIYBCABKAsyKC5zYWFzLmFjY291bnRzLnYxLkdpdEh1YkRhdGFzb3VyY2VDb25maWdIABI0CgNhcGkYBSABKAsyJS5zYWFzLmFjY291bnRzLnYxLkFwaURhdGFzb3VyY2VDb25maWdIABI8CgdjcmF3bGVyGAggASgLMikuc2Fhcy5hY2NvdW50cy52MS5DcmF3bGVyRGF0YXNvdXJjZUNvbmZpZ0gAEjoKBnVwbG9hZBgJIAEoCzIoLnNhYXMuYWNjb3VudHMudjEuVXBsb2FkRGF0YXNvdXJjZUNvbmZpZ0gAEiQKEGJvdW5kYXJ5X25vZGVfaWQYDCABKAlCCLpIBXIDsAEBSAESJgoQY29sbGVjdGlvbl9sYWJlbBgNIAEoCUIKukgHcgUQARj/AUgBEhwKCmNyZWRlbnRpYWwYBiABKAlCCLpIBXIDGIAgEiAKDndlYmhvb2tfc2VjcmV0GAcgASgJQgi6SAVyAxiACBImChRvYXV0aDJfY2xpZW50X3NlY3JldBgKIAEoCUIIukgFcgMYgAhCCAoGY29uZmlnQhEKCGJvdW5kYXJ5EgW6SAIIAUoECAMQBFIRdGFyZ2V0X2NvbGxlY3Rpb24iRQoRQWRkU291cmNlUmVzcG9uc2USMAoKZGF0YXNvdXJjZRgBIAEoCzIcLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZSKKAgoTVXBkYXRlU291cmNlUmVxdWVzdBIYCgZvcmdfaWQYASABKAlCCLpIBXIDsAEBEhQKAmlkGAIgASgJQgi6SAVyA7ABARIYCgZicmFuY2gYAyABKAlCCLpIBXIDGP8BEiAKBXBhdGhzGAQgAygJQhG6SA6SAQsQQCIHcgUQARiABBJJCg9maWxlX2V4dGVuc2lvbnMYBSADKAlCMLpILZIBKhAgIiZyJBACGCEyHl5cLltBLVphLXowLTldW0EtWmEtejAtOS5fLV0qJBI8CghleHBlY3RlZBgGIAEoCzIiLnNhYXMuYWNjb3VudHMudjEuU291cmNlRmV0Y2hTY29wZUIGukgDyAEBInkKEFNvdXJjZUZldGNoU2NvcGUSGAoGYnJhbmNoGAEgASgJQgi6SAVyAxj/ARIgCgVwYXRocxgCIAMoCUIRukgOkgELEEAiB3IFEAEYgAQSKQoPZmlsZV9leHRlbnNpb25zGAMgAygJQhC6SA2SAQoQICIGcgQQARghIlgKFFVwZGF0ZVNvdXJjZVJlc3BvbnNlEjAKCmRhdGFzb3VyY2UYASABKAsyHC5zYWFzLmFjY291bnRzLnYxLkRhdGFzb3VyY2USDgoGam9iX2lkGAIgASgJIloKFURhdGFzb3VyY2VDb25maWdGaWVsZBILCgNrZXkYASABKAkSFAoMZGlzcGxheV9uYW1lGAIgASgJEgwKBGhlbHAYAyABKAkSEAoIcmVxdWlyZWQYBCABKAgixAEKGURhdGFzb3VyY2VDb25uZWN0b3JCdWRnZXQSGgoSbWF4X2l0ZW1zX3Blcl9jYWxsGAEgASgNEhoKEm1heF9ieXRlc19wZXJfY2FsbBgCIAEoAxIWCg5tYXhfaXRlbV9ieXRlcxgDIAEoAxIdChVvcGVyYXRpb25zX3Blcl93aW5kb3cYBCABKA0SFgoOd2luZG93X3NlY29uZHMYBSABKAMSIAoYYmFja2dyb3VuZF9zaGFyZV9wZXJjZW50GAYgASgNIqoGChxEYXRhc291cmNlUHJvdmlkZXJEZXNjcmlwdG9yEjYKCHByb3ZpZGVyGAEgASgOMiQuc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlUHJvdmlkZXISFAoMZGlzcGxheV9uYW1lGAIgASgJEhMKC2Rlc2NyaXB0aW9uGAMgASgJEj4KDWNvbmZpZ19maWVsZHMYBCADKAsyJy5zYWFzLmFjY291bnRzLnYxLkRhdGFzb3VyY2VDb25maWdGaWVsZBIYChBzdXBwb3J0c193ZWJob29rGAUgASgIEkcKGnN1cHBvcnRlZF9jcmVkZW50aWFsX2tpbmRzGAYgAygOMiMuc2Fhcy5hY2NvdW50cy52MS5BcGlDcmVkZW50aWFsS2luZBIRCgljb25uZWN0b3IYByABKAkSQQoJaW50ZXJmYWNlGAggASgOMi4uc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlQ29ubmVjdG9ySW50ZXJmYWNlEkQKEGNyZWRlbnRpYWxfbW9kZXMYCSADKA4yKi5zYWFzLmFjY291bnRzLnYxLkRhdGFzb3VyY2VDcmVkZW50aWFsTW9kZRI/Cg1yZWFkZXJzX21vZGVsGAogASgOMiguc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlUmVhZGVyc01vZGVsEjsKBmJ1ZGdldBgLIAEoCzIrLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZUNvbm5lY3RvckJ1ZGdldBISCgpjb25mb3JtYW50GAwgASgIEhcKD2NvbmZvcm1hbmNlX2dhcBgNIAEoCRIbChNhY2NlcHRzX25ld19zb3VyY2VzGA4gASgIEiAKGGxpdmVfZGVsaXZlcnlfY29uZmlndXJlZBgPIAEoCBIaChJhY2NlcHRzX29wZXJhdGlvbnMYECABKAgSFgoOb3BlcmF0aW9uc19nYXAYESABKAkSGwoTb2F1dGhfYXV0aG9yaXplX3VybBgSIAEoCRIXCg9vYXV0aF90b2tlbl91cmwYEyABKAkSFAoMb2F1dGhfc2NvcGVzGBQgAygJIh0KG0dldERhdGFzb3VyY2VDYXRhbG9nUmVxdWVzdCJhChxHZXREYXRhc291cmNlQ2F0YWxvZ1Jlc3BvbnNlEkEKCXByb3ZpZGVycxgBIAMoCzIuLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZVByb3ZpZGVyRGVzY3JpcHRvciIuChJMaXN0U291cmNlc1JlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABASJIChNMaXN0U291cmNlc1Jlc3BvbnNlEjEKC2RhdGFzb3VyY2VzGAEgAygLMhwuc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlIkIKEEdldFNvdXJjZVJlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARIUCgJpZBgCIAEoCUIIukgFcgOwAQEiRQoRR2V0U291cmNlUmVzcG9uc2USMAoKZGF0YXNvdXJjZRgBIAEoCzIcLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZSJ2ChFTeW5jU291cmNlUmVxdWVzdBIYCgZvcmdfaWQYASABKAlCCLpIBXIDsAEBEhQKAmlkGAIgASgJQgi6SAVyA7ABARIeCgxhY2Nlc3NfdG9rZW4YAyABKAlCCLpIBXIDGIAgEhEKCXJlY29ubmVjdBgEIAEoCCIkChJTeW5jU291cmNlUmVzcG9uc2USDgoGam9iX2lkGAEgASgJImoKFEdldFNvdXJjZVN5bmNSZXF1ZXN0EhgKBm9yZ19pZBgBIAEoCUIIukgFcgOwAQESGwoJc291cmNlX2lkGAIgASgJQgi6SAVyA7ABARIbCgZqb2JfaWQYAyABKAlCC7pICNgBAXIDsAEBIpUBChJTb3VyY2VTeW5jRGVsaXZlcnkSGAoGam9iX2lkGAEgASgJQgi6SAVyA7ABARItCgVzdGF0ZRgCIAEoDjIWLnNhYXMuam9icy52MS5Kb2JTdGF0ZUIGukgDyAEBEjYKCWV4ZWN1dGlvbhgDIAEoCzIjLnNhYXMuam9icy52MS5Kb2JFeGVjdXRpb25SZWZlcmVuY2UiiwEKEVNvdXJjZVN5bmNDaGFuZ2VzEg0KBWZpbGVzGAEgASgNEg0KBWFkZGVkGAIgASgNEhAKCG1vZGlmaWVkGAMgASgNEg8KB2RlbGV0ZWQYBCABKA0SEwoLc3BsaXRfa25vd24YBSABKAgSEAoIc25hcHNob3QYBiABKAgSDgoGY29tbWl0GAcgASgJIq0BChFTb3VyY2VTeW5jRmFpbHVyZRI5CgZyZWFzb24YASABKA4yKS5zYWFzLmFjY291bnRzLnYxLlNvdXJjZVN5bmNGYWlsdXJlUmVhc29uEgwKBGNvZGUYAiABKAkSDwoHbWVzc2FnZRgDIAEoCRIQCghyZXRyeWluZxgEIAEoCBIsCghyZXRyeV9hdBgFIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAihAQKElNvdXJjZVN5bmNQcm9ncmVzcxIwCgVwaGFzZRgBIAEoDjIhLnNhYXMuYWNjb3VudHMudjEuU291cmNlU3luY1BoYXNlEjQKB3RyaWdnZXIYAiABKA4yIy5zYWFzLmFjY291bnRzLnYxLlNvdXJjZVN5bmNUcmlnZ2VyEi0KCXF1ZXVlZF9hdBgDIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASLwoLZmV0Y2hpbmdfYXQYBCABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEi8KC2NvbXBpbGVkX2F0GAUgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIxCg1oYW5kZWRfb2ZmX2F0GAYgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIvCgtmaW5pc2hlZF9hdBgHIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASNAoHY2hhbmdlcxgIIAEoCzIjLnNhYXMuYWNjb3VudHMudjEuU291cmNlU3luY0NoYW5nZXMSNAoHZmFpbHVyZRgJIAEoCzIjLnNhYXMuYWNjb3VudHMudjEuU291cmNlU3luY0ZhaWx1cmUSDwoHYXR0ZW1wdBgKIAEoDRIUCgxtYXhfYXR0ZW1wdHMYCyABKA0i0gEKFUdldFNvdXJjZVN5bmNSZXNwb25zZRIYCgZqb2JfaWQYASABKAlCCLpIBXIDsAEBEi0KBXN0YXRlGAIgASgOMhYuc2Fhcy5qb2JzLnYxLkpvYlN0YXRlQga6SAPIAQESOAoKZGVsaXZlcmllcxgDIAMoCzIkLnNhYXMuYWNjb3VudHMudjEuU291cmNlU3luY0RlbGl2ZXJ5EjYKCHByb2dyZXNzGAQgASgLMiQuc2Fhcy5hY2NvdW50cy52MS5Tb3VyY2VTeW5jUHJvZ3Jlc3MiRQoTRGVsZXRlU291cmNlUmVxdWVzdBIYCgZvcmdfaWQYASABKAlCCLpIBXIDsAEBEhQKAmlkGAIgASgJQgi6SAVyA7ABASIWChREZWxldGVTb3VyY2VSZXNwb25zZSJWChNHaXRIdWJBcHBSZXBvc2l0b3J5EgwKBHJlcG8YASABKAkSFgoOZGVmYXVsdF9icmFuY2gYAiABKAkSGQoRYWxyZWFkeV9jb25uZWN0ZWQYAyABKAgiNgoaQmVnaW5HaXRIdWJBcHBTZXR1cFJlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABASJxChtCZWdpbkdpdEh1YkFwcFNldHVwUmVzcG9uc2USEwoLaW5zdGFsbF91cmwYASABKAkSDQoFc3RhdGUYAiABKAkSLgoKZXhwaXJlc19hdBgDIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAinAEKHUNvbXBsZXRlR2l0SHViQXBwU2V0dXBSZXF1ZXN0EhgKBm9yZ19pZBgBIAEoCUIIukgFcgOwAQESGQoFc3RhdGUYAiABKAlCCrpIB3IFEAEY/wESLAoPaW5zdGFsbGF0aW9uX2lkGAMgASgJQhO6SBByDhABGCAyCF5bMC05XSskEhgKBGNvZGUYBCABKAlCCrpIB3IFEAEY/wEidgoeQ29tcGxldGVHaXRIdWJBcHBTZXR1cFJlc3BvbnNlEhcKD2luc3RhbGxhdGlvbl9pZBgBIAEoCRI7CgxyZXBvc2l0b3JpZXMYAiADKAsyJS5zYWFzLmFjY291bnRzLnYxLkdpdEh1YkFwcFJlcG9zaXRvcnkiUQofTWlncmF0ZUdpdEh1YlNvdXJjZVRvQXBwUmVxdWVzdBIYCgZvcmdfaWQYASABKAlCCLpIBXIDsAEBEhQKAmlkGAIgASgJQgi6SAVyA7ABASJUCiBNaWdyYXRlR2l0SHViU291cmNlVG9BcHBSZXNwb25zZRIwCgpkYXRhc291cmNlGAEgASgLMhwuc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlIsQBChVEYXRhc291cmNlQWNjb3VudExpbmsSCgoCaWQYASABKAkSDgoGb3JnX2lkGAIgASgJEg8KB3VzZXJfaWQYAyABKAkSEQoJY29ubmVjdG9yGAQgASgJEhsKE3Byb3ZpZGVyX2FjY291bnRfaWQYBSABKAkSHgoWcHJvdmlkZXJfYWNjb3VudF9sb2dpbhgGIAEoCRIuCgpjcmVhdGVkX2F0GAcgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCK3AQoWRGF0YXNvdXJjZUdyb3VwQmluZGluZxIKCgJpZBgBIAEoCRIOCgZvcmdfaWQYAiABKAkSEQoJY29ubmVjdG9yGAMgASgJEhkKEXByb3ZpZGVyX2dyb3VwX2lkGAQgASgJEg8KB3RlYW1faWQYBSABKAkSEgoKY3JlYXRlZF9ieRgGIAEoCRIuCgpjcmVhdGVkX2F0GAcgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCKUAgoYRGF0YXNvdXJjZVZlcmlmaWVkRG9tYWluEgoKAmlkGAEgASgJEg4KBm9yZ19pZBgCIAEoCRIOCgZkb21haW4YAyABKAkSOAoGc3RhdHVzGAQgASgOMiguc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlRG9tYWluU3RhdHVzEhcKD3R4dF9yZWNvcmRfbmFtZRgFIAEoCRIYChB0eHRfcmVjb3JkX3ZhbHVlGAYgASgJEi8KC3ZlcmlmaWVkX2F0GAcgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIuCgpjcmVhdGVkX2F0GAggASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCIzChdEYXRhc291cmNlRGlyZWN0b3J5VGVhbRIKCgJpZBgBIAEoCRIMCgRuYW1lGAIgASgJIqYBCiFCZWdpbkRhdGFzb3VyY2VBY2NvdW50TGlua1JlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARI0Cgljb25uZWN0b3IYAiABKAlCIbpIHnIcEAIYIDIWXlthLXpdW2EtejAtOS1dezEsMzF9JBIeCgxyZWRpcmVjdF91cmkYAyABKAlCCLpIBXIDGIAQEhEKCXNvdXJjZV9pZBgEIAEoCSJ6CiJCZWdpbkRhdGFzb3VyY2VBY2NvdW50TGlua1Jlc3BvbnNlEhUKDWF1dGhvcml6ZV91cmwYASABKAkSDQoFc3RhdGUYAiABKAkSLgoKZXhwaXJlc19hdBgDIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAidQokQ29tcGxldGVEYXRhc291cmNlQWNjb3VudExpbmtSZXF1ZXN0EhgKBm9yZ19pZBgBIAEoCUIIukgFcgOwAQESGQoFc3RhdGUYAiABKAlCCrpIB3IFEAEYgEASGAoEY29kZRgDIAEoCUIKukgHcgUQARj/ASJeCiVDb21wbGV0ZURhdGFzb3VyY2VBY2NvdW50TGlua1Jlc3BvbnNlEjUKBGxpbmsYASABKAsyJy5zYWFzLmFjY291bnRzLnYxLkRhdGFzb3VyY2VBY2NvdW50TGluayI/CiNMaXN0TXlEYXRhc291cmNlQWNjb3VudExpbmtzUmVxdWVzdBIYCgZvcmdfaWQYASABKAlCCLpIBXIDsAEBIl4KJExpc3RNeURhdGFzb3VyY2VBY2NvdW50TGlua3NSZXNwb25zZRI2CgVsaW5rcxgBIAMoCzInLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZUFjY291bnRMaW5rIlQKIkRlbGV0ZURhdGFzb3VyY2VBY2NvdW50TGlua1JlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARIUCgJpZBgCIAEoCUIIukgFcgOwAQEiJQojRGVsZXRlRGF0YXNvdXJjZUFjY291bnRMaW5rUmVzcG9uc2UiOQodR2V0RGF0YXNvdXJjZURpcmVjdG9yeVJlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABASKLAgoeR2V0RGF0YXNvdXJjZURpcmVjdG9yeVJlc3BvbnNlEjYKBWxpbmtzGAEgAygLMicuc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlQWNjb3VudExpbmsSOgoIYmluZGluZ3MYAiADKAsyKC5zYWFzLmFjY291bnRzLnYxLkRhdGFzb3VyY2VHcm91cEJpbmRpbmcSOwoHZG9tYWlucxgDIAMoCzIqLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZVZlcmlmaWVkRG9tYWluEjgKBXRlYW1zGAQgAygLMikuc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlRGlyZWN0b3J5VGVhbSKuAQoaQmluZERhdGFzb3VyY2VHcm91cFJlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARI0Cgljb25uZWN0b3IYAiABKAlCIbpIHnIcEAIYIDIWXlthLXpdW2EtejAtOS1dezEsMzF9JBIlChFwcm92aWRlcl9ncm91cF9pZBgDIAEoCUIKukgHcgUQARj/ARIZCgd0ZWFtX2lkGAQgASgJQgi6SAVyA7ABASJYChtCaW5kRGF0YXNvdXJjZUdyb3VwUmVzcG9uc2USOQoHYmluZGluZxgBIAEoCzIoLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZUdyb3VwQmluZGluZyJOChxVbmJpbmREYXRhc291cmNlR3JvdXBSZXF1ZXN0EhgKBm9yZ19pZBgBIAEoCUIIukgFcgOwAQESFAoCaWQYAiABKAlCCLpIBXIDsAEBIh8KHVVuYmluZERhdGFzb3VyY2VHcm91cFJlc3BvbnNlIlQKHENsYWltRGF0YXNvdXJjZURvbWFpblJlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARIaCgZkb21haW4YAiABKAlCCrpIB3IFEAMY/QEiWwodQ2xhaW1EYXRhc291cmNlRG9tYWluUmVzcG9uc2USOgoGZG9tYWluGAEgASgLMiouc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlVmVyaWZpZWREb21haW4iTwodVmVyaWZ5RGF0YXNvdXJjZURvbWFpblJlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARIUCgJpZBgCIAEoCUIIukgFcgOwAQEiXAoeVmVyaWZ5RGF0YXNvdXJjZURvbWFpblJlc3BvbnNlEjoKBmRvbWFpbhgBIAEoCzIqLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZVZlcmlmaWVkRG9tYWluIk8KHURlbGV0ZURhdGFzb3VyY2VEb21haW5SZXF1ZXN0EhgKBm9yZ19pZBgBIAEoCUIIukgFcgOwAQESFAoCaWQYAiABKAlCCLpIBXIDsAEBIiAKHkRlbGV0ZURhdGFzb3VyY2VEb21haW5SZXNwb25zZSKeAgoQU291cmNlRGVsZWdhdGlvbhIKCgJpZBgBIAEoCRIRCglzb3VyY2VfaWQYAiABKAkSFAoMcHJpbmNpcGFsX2lkGAMgASgJEg4KBm1vZHVsZRgEIAEoCRIPCgdiaW5kaW5nGAUgASgJEi4KCmNyZWF0ZWRfYXQYBiABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEi4KCnJldm9rZWRfYXQYByABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEkAKCnJldm9jYXRpb24YCCABKA4yLC5zYWFzLmFjY291bnRzLnYxLlNvdXJjZURlbGVnYXRpb25SZXZvY2F0aW9uEhIKCnJldm9rZWRfYnkYCSABKAkigQEKHExpc3RTb3VyY2VEZWxlZ2F0aW9uc1JlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARIgCglzb3VyY2VfaWQYAiABKAlCCLpIBXIDsAEBSACIAQESFwoPaW5jbHVkZV9yZXZva2VkGAMgASgIQgwKCl9zb3VyY2VfaWQiWAodTGlzdFNvdXJjZURlbGVnYXRpb25zUmVzcG9uc2USNwoLZGVsZWdhdGlvbnMYASADKAsyIi5zYWFzLmFjY291bnRzLnYxLlNvdXJjZURlbGVnYXRpb24iTwodUmV2b2tlU291cmNlRGVsZWdhdGlvblJlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARIUCgJpZBgCIAEoCUIIukgFcgOwAQEiWAoeUmV2b2tlU291cmNlRGVsZWdhdGlvblJlc3BvbnNlEjYKCmRlbGVnYXRpb24YASABKAsyIi5zYWFzLmFjY291bnRzLnYxLlNvdXJjZURlbGVnYXRpb24iMgoPVHJpZ2dlcklkZW50aXR5Eg4KBnNvdXJjZRgBIAEoCRIPCgdiaW5kaW5nGAIgASgJIoQBChdEZWNsYXJlZFRyaWdnZXJTY2hlZHVsZRIMCgRjcm9uGAEgASgJEhAKCHRpbWV6b25lGAIgASgJEhIKCm1pc3NlZF9ydW4YAyABKAkSDwoHb3ZlcmxhcBgEIAEoCRIQCghkZWFkbGluZRgFIAEoAxISCgppbnB1dF9qc29uGAYgASgJIucCChpUcmlnZ2VyRGVsZWdhdGlvblJlZmVyZW5jZRIRCglyZWZlcmVuY2UYASABKAkSFQoNZGVsZWdhdGlvbl9pZBgCIAEoCRIOCgZtb2R1bGUYAyABKAkSDwoHYmluZGluZxgEIAEoCRIQCghhdWRpZW5jZRgFIAEoCRIOCgZ0ZW5hbnQYBiABKAkSGgoSb3duZXJfcHJpbmNpcGFsX2lkGAcgASgJEg4KBnNjb3BlcxgIIAMoCRIOCgZhY3RpdmUYCSABKAgSLgoKY3JlYXRlZF9hdBgKIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASLgoKcmV2b2tlZF9hdBgLIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASQAoKcmV2b2NhdGlvbhgMIAEoDjIsLnNhYXMuYWNjb3VudHMudjEuU291cmNlRGVsZWdhdGlvblJldm9jYXRpb24ivgEKE1RyaWdnZXJDaGlsZENvbnRleHQSFQoNZGVsZWdhdGlvbl9pZBgBIAEoCRIOCgZtb2R1bGUYAiABKAkSDwoHYmluZGluZxgDIAEoCRIQCghhdWRpZW5jZRgEIAEoCRIOCgZzY29wZXMYBSADKAkSDgoGbG9va3VwGAYgASgIEi0KCW1pbnRlZF9hdBgHIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASDgoGcmVjb3JkGAggASgJIs0CChhTb3VyY2VUcmlnZ2VyRGVjbGFyYXRpb24SDwoHcmVsZWFzZRgBIAEoCRIRCglvcGVyYXRpb24YAiABKAkSOwoIc2NoZWR1bGUYAyABKAsyKS5zYWFzLmFjY291bnRzLnYxLkRlY2xhcmVkVHJpZ2dlclNjaGVkdWxlEjMKCGlkZW50aXR5GAQgASgLMiEuc2Fhcy5hY2NvdW50cy52MS5UcmlnZ2VySWRlbnRpdHkSDgoGZGlnZXN0GAUgASgJEjYKEmxhc3Rfb2NjdXJyZW5jZV9hdBgGIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASNgoSbmV4dF9vY2N1cnJlbmNlX2F0GAcgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIbChN1bmRlY2xhcmFibGVfcmVhc29uGAggASgJIlYKHURlbGVnYXRlU291cmNlVHJpZ2dlcnNSZXF1ZXN0EhgKBm9yZ19pZBgBIAEoCUIIukgFcgOwAQESGwoJc291cmNlX2lkGAIgASgJQgi6SAVyA7ABASJjCh5EZWxlZ2F0ZVNvdXJjZVRyaWdnZXJzUmVzcG9uc2USQQoLZGVsZWdhdGlvbnMYASADKAsyLC5zYWFzLmFjY291bnRzLnYxLlRyaWdnZXJEZWxlZ2F0aW9uUmVmZXJlbmNlIlsKIkdldFNvdXJjZVRyaWdnZXJEZWNsYXJhdGlvblJlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARIbCglzb3VyY2VfaWQYAiABKAlCCLpIBXIDsAEBIuIBCiNHZXRTb3VyY2VUcmlnZ2VyRGVjbGFyYXRpb25SZXNwb25zZRI/CgtkZWNsYXJhdGlvbhgBIAEoCzIqLnNhYXMuYWNjb3VudHMudjEuU291cmNlVHJpZ2dlckRlY2xhcmF0aW9uEkEKC2RlbGVnYXRpb25zGAIgAygLMiwuc2Fhcy5hY2NvdW50cy52MS5UcmlnZ2VyRGVsZWdhdGlvblJlZmVyZW5jZRI3CghjaGlsZHJlbhgDIAMoCzIlLnNhYXMuYWNjb3VudHMudjEuVHJpZ2dlckNoaWxkQ29udGV4dCJuChZSZWNvbmNpbGVTb3VyY2VSZXF1ZXN0EhgKBm9yZ19pZBgBIAEoCUIIukgFcgOwAQESGwoJc291cmNlX2lkGAIgASgJQgi6SAVyA7ABARIdCgllZmZlY3RfaWQYAyABKAlCCrpIB3IFEAEYgAEiTAoXUmVjb25jaWxlU291cmNlUmVzcG9uc2USEQoJZWZmZWN0X2lkGAEgASgJEg4KBnN0YXR1cxgCIAEoCRIOCgZqb2JfaWQYAyABKAkiPQocTG9va3VwUmVjb25jaWxlU291cmNlUmVxdWVzdBIdCgllZmZlY3RfaWQYASABKAlCCrpIB3IFEAEYgAEi8QIKD1NvdXJjZU9wZXJhdGlvbhIMCgRuYW1lGAEgASgJEhMKC2Rlc2NyaXB0aW9uGAIgASgJEg4KBm1ldGhvZBgDIAEoCRIMCgRwYXRoGAQgASgJEg0KBXF1ZXJ5GAUgAygJEi0KDGlucHV0X3NjaGVtYRgGIAEoCzIXLmdvb2dsZS5wcm90b2J1Zi5TdHJ1Y3QSLgoNb3V0cHV0X3NjaGVtYRgHIAEoCzIXLmdvb2dsZS5wcm90b2J1Zi5TdHJ1Y3QSOAoGZWZmZWN0GAggASgOMiguc2Fhcy5hY2NvdW50cy52MS5Tb3VyY2VPcGVyYXRpb24uRWZmZWN0EhgKEG1heF9vdXRwdXRfYnl0ZXMYCSABKA0SDgoGZGlnZXN0GAogASgJIksKBkVmZmVjdBIWChJFRkZFQ1RfVU5TUEVDSUZJRUQQABIUChBFRkZFQ1RfUkVBRF9PTkxZEAESEwoPRUZGRUNUX01VVEFUSU9OEAIijgEKHkRlY2xhcmVTb3VyY2VPcGVyYXRpb25zUmVxdWVzdBIYCgZvcmdfaWQYASABKAlCCLpIBXIDsAEBEhsKCXNvdXJjZV9pZBgCIAEoCUIIukgFcgOwAQESNQoKb3BlcmF0aW9ucxgDIAMoCzIhLnNhYXMuYWNjb3VudHMudjEuU291cmNlT3BlcmF0aW9uIlgKH0RlY2xhcmVTb3VyY2VPcGVyYXRpb25zUmVzcG9uc2USNQoKb3BlcmF0aW9ucxgBIAMoCzIhLnNhYXMuYWNjb3VudHMudjEuU291cmNlT3BlcmF0aW9uIlQKG0xpc3RTb3VyY2VPcGVyYXRpb25zUmVxdWVzdBIYCgZvcmdfaWQYASABKAlCCLpIBXIDsAEBEhsKCXNvdXJjZV9pZBgCIAEoCUIIukgFcgOwAQEiVQocTGlzdFNvdXJjZU9wZXJhdGlvbnNSZXNwb25zZRI1CgpvcGVyYXRpb25zGAEgAygLMiEuc2Fhcy5hY2NvdW50cy52MS5Tb3VyY2VPcGVyYXRpb24irgIKHEludm9rZVNvdXJjZU9wZXJhdGlvblJlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARIbCglzb3VyY2VfaWQYAiABKAlCCLpIBXIDsAEBEjAKCW9wZXJhdGlvbhgDIAEoCUIdukgachgyFl5bYS16XVthLXowLTlfXXswLDYzfSQSHQoKaW5wdXRfanNvbhgEIAEoCUIJukgGcgQogIAEEhsKCWVmZmVjdF9pZBgFIAEoCUIIukgFcgMogAESSQobZXhwZWN0ZWRfZGVjbGFyYXRpb25fZGlnZXN0GAYgASgJQh+6SBxyGjIYXihzaGEyNTY6WzAtOWEtZl17NjR9KT8kSACIAQFCHgocX2V4cGVjdGVkX2RlY2xhcmF0aW9uX2RpZ2VzdCJ/ChZTb3VyY2VPcGVyYXRpb25SZWNlaXB0EhEKCWVmZmVjdF9pZBgBIAEoCRIUCgxjb21taXR0ZWRfYXQYAiABKAkSDgoGc3RhdHVzGAMgASgJEhcKD3Byb3ZpZGVyX3N0YXR1cxgEIAEoDRITCgtvdXRwdXRfanNvbhgFIAEoCSJvCh1JbnZva2VTb3VyY2VPcGVyYXRpb25SZXNwb25zZRITCgtvdXRwdXRfanNvbhgBIAEoCRI5CgdyZWNlaXB0GAIgASgLMiguc2Fhcy5hY2NvdW50cy52MS5Tb3VyY2VPcGVyYXRpb25SZWNlaXB0IkMKIkxvb2t1cEludm9rZVNvdXJjZU9wZXJhdGlvblJlcXVlc3QSHQoJZWZmZWN0X2lkGAEgASgJQgq6SAdyBRABKIABInkKI1BydW5lU291cmNlT3BlcmF0aW9uUmVjZWlwdHNSZXF1ZXN0EhgKBm9yZ19pZBgBIAEoCUIIukgFcgOwAQESGwoJc291cmNlX2lkGAIgASgJQgi6SAVyA7ABARIbCgllZmZlY3RfaWQYAyABKAlCCLpIBXIDKIABIlsKJFBydW5lU291cmNlT3BlcmF0aW9uUmVjZWlwdHNSZXNwb25zZRIRCgllZmZlY3RfaWQYASABKAkSEAoIcmVjZWlwdHMYAiABKAMSDgoGc3RhdHVzGAMgASgJIkoKKUxvb2t1cFBydW5lU291cmNlT3BlcmF0aW9uUmVjZWlwdHNSZXF1ZXN0Eh0KCWVmZmVjdF9pZBgBIAEoCUIKukgHcgUQASiAASq3AQoSRGF0YXNvdXJjZVByb3ZpZGVyEiMKH0RBVEFTT1VSQ0VfUFJPVklERVJfVU5TUEVDSUZJRUQQABIeChpEQVRBU09VUkNFX1BST1ZJREVSX0dJVEhVQhABEhsKF0RBVEFTT1VSQ0VfUFJPVklERVJfQVBJEAISHwobREFUQVNPVVJDRV9QUk9WSURFUl9DUkFXTEVSEAMSHgoaREFUQVNPVVJDRV9QUk9WSURFUl9VUExPQUQQBCqRAQoQRGF0YXNvdXJjZVN0YXR1cxIhCh1EQVRBU09VUkNFX1NUQVRVU19VTlNQRUNJRklFRBAAEhwKGERBVEFTT1VSQ0VfU1RBVFVTX0FDVElWRRABEhwKGERBVEFTT1VSQ0VfU1RBVFVTX1BBVVNFRBACEh4KGkRBVEFTT1VSQ0VfU1RBVFVTX0RFR1JBREVEEAMqvAEKFkRhdGFzb3VyY2VMaXZlRGVsaXZlcnkSKAokREFUQVNPVVJDRV9MSVZFX0RFTElWRVJZX1VOU1BFQ0lGSUVEEAASIQodREFUQVNPVVJDRV9MSVZFX0RFTElWRVJZX05PTkUQARIrCidEQVRBU09VUkNFX0xJVkVfREVMSVZFUllfU09VUkNFX1dFQkhPT0sQAhIoCiREQVRBU09VUkNFX0xJVkVfREVMSVZFUllfQVBQX1dFQkhPT0sQAyrWAQoRQXBpQ3JlZGVudGlhbEtpbmQSIwofQVBJX0NSRURFTlRJQUxfS0lORF9VTlNQRUNJRklFRBAAEh4KGkFQSV9DUkVERU5USUFMX0tJTkRfQkVBUkVSEAESHQoZQVBJX0NSRURFTlRJQUxfS0lORF9CQVNJQxACEh4KGkFQSV9DUkVERU5USUFMX0tJTkRfSEVBREVSEAMSHQoZQVBJX0NSRURFTlRJQUxfS0lORF9RVUVSWRAEEh4KGkFQSV9DUkVERU5USUFMX0tJTkRfT0FVVEgyEAUqpgIKHERhdGFzb3VyY2VDb25uZWN0b3JJbnRlcmZhY2USLgoqREFUQVNPVVJDRV9DT05ORUNUT1JfSU5URVJGQUNFX1VOU1BFQ0lGSUVEEAASKAokREFUQVNPVVJDRV9DT05ORUNUT1JfSU5URVJGQUNFX0ZJTEVTEAESKAokREFUQVNPVVJDRV9DT05ORUNUT1JfSU5URVJGQUNFX1BBR0VTEAISKgomREFUQVNPVVJDRV9DT05ORUNUT1JfSU5URVJGQUNFX1JFQ09SRFMQAxIrCidEQVRBU09VUkNFX0NPTk5FQ1RPUl9JTlRFUkZBQ0VfTUVTU0FHRVMQBBIpCiVEQVRBU09VUkNFX0NPTk5FQ1RPUl9JTlRFUkZBQ0VfRVZFTlRTEAUq7AEKGERhdGFzb3VyY2VDcmVkZW50aWFsTW9kZRIqCiZEQVRBU09VUkNFX0NSRURFTlRJQUxfTU9ERV9VTlNQRUNJRklFRBAAEiMKH0RBVEFTT1VSQ0VfQ1JFREVOVElBTF9NT0RFX05PTkUQARImCiJEQVRBU09VUkNFX0NSRURFTlRJQUxfTU9ERV9PUkdfQVBQEAISKQolREFUQVNPVVJDRV9DUkVERU5USUFMX01PREVfVVNFUl9PQVVUSBADEiwKKERBVEFTT1VSQ0VfQ1JFREVOVElBTF9NT0RFX1NUQVRJQ19TRUNSRVQQBCqXAQoWRGF0YXNvdXJjZVJlYWRlcnNNb2RlbBIoCiREQVRBU09VUkNFX1JFQURFUlNfTU9ERUxfVU5TUEVDSUZJRUQQABIqCiZEQVRBU09VUkNFX1JFQURFUlNfTU9ERUxfU09VUkNFX1NDT1BFRBABEicKI0RBVEFTT1VSQ0VfUkVBREVSU19NT0RFTF9UUkFOU0xBVEVEEAIq7gEKD1NvdXJjZVN5bmNQaGFzZRIhCh1TT1VSQ0VfU1lOQ19QSEFTRV9VTlNQRUNJRklFRBAAEhwKGFNPVVJDRV9TWU5DX1BIQVNFX1FVRVVFRBABEh4KGlNPVVJDRV9TWU5DX1BIQVNFX0ZFVENISU5HEAISHgoaU09VUkNFX1NZTkNfUEhBU0VfQ09NUElMRUQQAxIgChxTT1VSQ0VfU1lOQ19QSEFTRV9IQU5ERURfT0ZGEAQSGgoWU09VUkNFX1NZTkNfUEhBU0VfRE9ORRAFEhwKGFNPVVJDRV9TWU5DX1BIQVNFX0ZBSUxFRBAGKqYDChdTb3VyY2VTeW5jRmFpbHVyZVJlYXNvbhIqCiZTT1VSQ0VfU1lOQ19GQUlMVVJFX1JFQVNPTl9VTlNQRUNJRklFRBAAEisKJ1NPVVJDRV9TWU5DX0ZBSUxVUkVfUkVBU09OX1JBVEVfTElNSVRFRBABEikKJVNPVVJDRV9TWU5DX0ZBSUxVUkVfUkVBU09OX0NSRURFTlRJQUwQAhIsCihTT1VSQ0VfU1lOQ19GQUlMVVJFX1JFQVNPTl9BQ0NFU1NfREVOSUVEEAMSKAokU09VUkNFX1NZTkNfRkFJTFVSRV9SRUFTT05fTk9UX0ZPVU5EEAQSKAokU09VUkNFX1NZTkNfRkFJTFVSRV9SRUFTT05fVE9PX0xBUkdFEAUSLworU09VUkNFX1NZTkNfRkFJTFVSRV9SRUFTT05fSE9TVF9VTkFWQUlMQUJMRRAGEiQKIFNPVVJDRV9TWU5DX0ZBSUxVUkVfUkVBU09OX09USEVSEAcSLgoqU09VUkNFX1NZTkNfRkFJTFVSRV9SRUFTT05fREVMSVZFUllfRkFJTEVEEAgqwgEKEVNvdXJjZVN5bmNUcmlnZ2VyEiMKH1NPVVJDRV9TWU5DX1RSSUdHRVJfVU5TUEVDSUZJRUQQABIeChpTT1VSQ0VfU1lOQ19UUklHR0VSX01BTlVBTBABEiEKHVNPVVJDRV9TWU5DX1RSSUdHRVJfU0NIRURVTEVEEAISHwobU09VUkNFX1NZTkNfVFJJR0dFUl9XRUJIT09LEAMSJAogU09VUkNFX1NZTkNfVFJJR0dFUl9SRUNPTkZJR1VSRUQQBCqPAQoWRGF0YXNvdXJjZURvbWFpblN0YXR1cxIoCiREQVRBU09VUkNFX0RPTUFJTl9TVEFUVVNfVU5TUEVDSUZJRUQQABIkCiBEQVRBU09VUkNFX0RPTUFJTl9TVEFUVVNfUEVORElORxABEiUKIURBVEFTT1VSQ0VfRE9NQUlOX1NUQVRVU19WRVJJRklFRBACKpUDChpTb3VyY2VEZWxlZ2F0aW9uUmV2b2NhdGlvbhIsCihTT1VSQ0VfREVMRUdBVElPTl9SRVZPQ0FUSU9OX1VOU1BFQ0lGSUVEEAASKAokU09VUkNFX0RFTEVHQVRJT05fUkVWT0NBVElPTl9SRVZPS0VEEAESKQolU09VUkNFX0RFTEVHQVRJT05fUkVWT0NBVElPTl9SRVBMQUNFRBACEi8KK1NPVVJDRV9ERUxFR0FUSU9OX1JFVk9DQVRJT05fU09VUkNFX0RFTEVURUQQAxIvCitTT1VSQ0VfREVMRUdBVElPTl9SRVZPQ0FUSU9OX01FTUJFUl9SRU1PVkVEEAQSMAosU09VUkNFX0RFTEVHQVRJT05fUkVWT0NBVElPTl9QRVJNSVNTSU9OX0xPU1QQBRIuCipTT1VSQ0VfREVMRUdBVElPTl9SRVZPQ0FUSU9OX1VTRVJfSU5BQ1RJVkUQBhIwCixTT1VSQ0VfREVMRUdBVElPTl9SRVZPQ0FUSU9OX0JJTkRJTkdfQ0hBTkdFRBAHMrE5ChFEYXRhc291cmNlU2VydmljZRK5AwocUHJ1bmVTb3VyY2VPcGVyYXRpb25SZWNlaXB0cxI1LnNhYXMuYWNjb3VudHMudjEuUHJ1bmVTb3VyY2VPcGVyYXRpb25SZWNlaXB0c1JlcXVlc3QaNi5zYWFzLmFjY291bnRzLnYxLlBydW5lU291cmNlT3BlcmF0aW9uUmVjZWlwdHNSZXNwb25zZSKpAsLzGGwIAhADGhFkYXRhc291cmNlOmludm9rZSIRZGF0YXNvdXJjZTppbnZva2UqDwoJc291cmNlX2lkEAQYAzABOiMKH3NhYXMuZGF0YXNvdXJjZS5yZWNlaXB0cy5wcnVuZWQQAkADSARQA1gDYAHa8xi0AQoCCDwSAwjYBBgFIgIIAioLVU5BVkFJTEFCTEUqElJFU09VUkNFX0VYSEFVU1RFRCoHQUJPUlRFRDIPc2Fhcy1kYXRhc291cmNlSkYvc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlU2VydmljZS9Mb29rdXBQcnVuZVNvdXJjZU9wZXJhdGlvblJlY2VpcHRzUIAgWIAgYAFyGAoGc291cmNlEgZpbnZva2USBHJlYWQYARLmAQoiTG9va3VwUHJ1bmVTb3VyY2VPcGVyYXRpb25SZWNlaXB0cxI7LnNhYXMuYWNjb3VudHMudjEuTG9va3VwUHJ1bmVTb3VyY2VPcGVyYXRpb25SZWNlaXB0c1JlcXVlc3QaNi5zYWFzLmFjY291bnRzLnYxLlBydW5lU291cmNlT3BlcmF0aW9uUmVjZWlwdHNSZXNwb25zZSJLwvMYRwgCEAMaD2RhdGFzb3VyY2U6cmVhZCIPZGF0YXNvdXJjZTpyZWFkKg8KCWVmZmVjdF9pZBAEGAMwAToCEAFAAUgDUANYA2ABEvwDChVJbnZva2VTb3VyY2VPcGVyYXRpb24SLi5zYWFzLmFjY291bnRzLnYxLkludm9rZVNvdXJjZU9wZXJhdGlvblJlcXVlc3QaLy5zYWFzLmFjY291bnRzLnYxLkludm9rZVNvdXJjZU9wZXJhdGlvblJlc3BvbnNlIoEDwvMYbggCEAMaEWRhdGFzb3VyY2U6aW52b2tlIhFkYXRhc291cmNlOmludm9rZSoPCglzb3VyY2VfaWQQBBgDMAE6JQohc2Fhcy5kYXRhc291cmNlLm9wZXJhdGlvbi5pbnZva2VkEARAA0gEUARYBGAB2vMYrwEKAgg8EgMI2AQYBSICCAIqC1VOQVZBSUxBQkxFKhJSRVNPVVJDRV9FWEhBVVNURUQqB0FCT1JURUQyD3NhYXMtZGF0YXNvdXJjZUo/L3NhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZVNlcnZpY2UvTG9va3VwSW52b2tlU291cmNlT3BlcmF0aW9uUICABFiAgEBgAXIYCgZzb3VyY2USBmludm9rZRIEcmVhZBgBgtPkkwJVOgEqIlAvdjEvb3JnYW5pemF0aW9ucy97b3JnX2lkfS9kYXRhc291cmNlcy97c291cmNlX2lkfS9vcGVyYXRpb25zL3tvcGVyYXRpb259Omludm9rZRKHAgobTG9va3VwSW52b2tlU291cmNlT3BlcmF0aW9uEjQuc2Fhcy5hY2NvdW50cy52MS5Mb29rdXBJbnZva2VTb3VyY2VPcGVyYXRpb25SZXF1ZXN0Gi8uc2Fhcy5hY2NvdW50cy52MS5JbnZva2VTb3VyY2VPcGVyYXRpb25SZXNwb25zZSKAAcLzGEcIAhADGg9kYXRhc291cmNlOnJlYWQiD2RhdGFzb3VyY2U6cmVhZCoPCgllZmZlY3RfaWQQBBgDMAE6AhABQAFIA1ADWARgAYLT5JMCLxItL3YxL2RhdGFzb3VyY2Utb3BlcmF0aW9uLXJlY2VpcHRzL3tlZmZlY3RfaWR9EoUCChRMaXN0U291cmNlT3BlcmF0aW9ucxItLnNhYXMuYWNjb3VudHMudjEuTGlzdFNvdXJjZU9wZXJhdGlvbnNSZXF1ZXN0Gi4uc2Fhcy5hY2NvdW50cy52MS5MaXN0U291cmNlT3BlcmF0aW9uc1Jlc3BvbnNlIo0BwvMYRAgCEAMaD2RhdGFzb3VyY2U6cmVhZCIPZGF0YXNvdXJjZTpyZWFkKgwKBm9yZ19pZBACGAEwAToCEAFAAUgDUANYA2ABgtPkkwI/Ej0vdjEvb3JnYW5pemF0aW9ucy97b3JnX2lkfS9kYXRhc291cmNlcy97c291cmNlX2lkfS9vcGVyYXRpb25zEpMCChdEZWNsYXJlU291cmNlT3BlcmF0aW9ucxIwLnNhYXMuYWNjb3VudHMudjEuRGVjbGFyZVNvdXJjZU9wZXJhdGlvbnNSZXF1ZXN0GjEuc2Fhcy5hY2NvdW50cy52MS5EZWNsYXJlU291cmNlT3BlcmF0aW9uc1Jlc3BvbnNlIpIBwvMYRggCEAQqDAoGb3JnX2lkEAIYATABOiYKInNhYXMuZGF0YXNvdXJjZS5vcGVyYXRpb24uZGVjbGFyZWQQAkABSARQA1gDYAGC0+STAkI6ASoiPS92MS9vcmdhbml6YXRpb25zL3tvcmdfaWR9L2RhdGFzb3VyY2VzL3tzb3VyY2VfaWR9L29wZXJhdGlvbnMSrAEKD0FkZEdpdEh1YlNvdXJjZRIoLnNhYXMuYWNjb3VudHMudjEuQWRkR2l0SHViU291cmNlUmVxdWVzdBopLnNhYXMuYWNjb3VudHMudjEuQWRkR2l0SHViU291cmNlUmVzcG9uc2UiRMLzGEAIAhAEKgwKBm9yZ19pZBACGAEwATogChxzYWFzLmRhdGFzb3VyY2Uuc291cmNlLmFkZGVkEAJAAUgEUARYA2ABEpoBCglBZGRTb3VyY2USIi5zYWFzLmFjY291bnRzLnYxLkFkZFNvdXJjZVJlcXVlc3QaIy5zYWFzLmFjY291bnRzLnYxLkFkZFNvdXJjZVJlc3BvbnNlIkTC8xhACAIQBCoMCgZvcmdfaWQQAhgBMAE6IAocc2Fhcy5kYXRhc291cmNlLnNvdXJjZS5hZGRlZBACQAFIBFAEWANgARKlAQoMVXBkYXRlU291cmNlEiUuc2Fhcy5hY2NvdW50cy52MS5VcGRhdGVTb3VyY2VSZXF1ZXN0GiYuc2Fhcy5hY2NvdW50cy52MS5VcGRhdGVTb3VyY2VSZXNwb25zZSJGwvMYQggCEAQqDAoGb3JnX2lkEAIYATABOiIKHnNhYXMuZGF0YXNvdXJjZS5zb3VyY2UudXBkYXRlZBACQAFIBFACWANgARKPAQoUR2V0RGF0YXNvdXJjZUNhdGFsb2cSLS5zYWFzLmFjY291bnRzLnYxLkdldERhdGFzb3VyY2VDYXRhbG9nUmVxdWVzdBouLnNhYXMuYWNjb3VudHMudjEuR2V0RGF0YXNvdXJjZUNhdGFsb2dSZXNwb25zZSIYwvMYFAgCEAEwAToCEAFAAUgDUAJYAmABEoIBCgtMaXN0U291cmNlcxIkLnNhYXMuYWNjb3VudHMudjEuTGlzdFNvdXJjZXNSZXF1ZXN0GiUuc2Fhcy5hY2NvdW50cy52MS5MaXN0U291cmNlc1Jlc3BvbnNlIibC8xgiCAIQAyoMCgZvcmdfaWQQAhgBMAE6AhABQAFIA1ACWANgARJ8CglHZXRTb3VyY2USIi5zYWFzLmFjY291bnRzLnYxLkdldFNvdXJjZVJlcXVlc3QaIy5zYWFzLmFjY291bnRzLnYxLkdldFNvdXJjZVJlc3BvbnNlIibC8xgiCAIQAyoMCgZvcmdfaWQQAhgBMAE6AhABQAFIA1ACWANgARKeAQoKU3luY1NvdXJjZRIjLnNhYXMuYWNjb3VudHMudjEuU3luY1NvdXJjZVJlcXVlc3QaJC5zYWFzLmFjY291bnRzLnYxLlN5bmNTb3VyY2VSZXNwb25zZSJFwvMYQQgCEAQqDAoGb3JnX2lkEAIYATABOiEKHXNhYXMuZGF0YXNvdXJjZS5zb3VyY2Uuc3luY2VkEAJAAUgEUARYAmABEogBCg1HZXRTb3VyY2VTeW5jEiYuc2Fhcy5hY2NvdW50cy52MS5HZXRTb3VyY2VTeW5jUmVxdWVzdBonLnNhYXMuYWNjb3VudHMudjEuR2V0U291cmNlU3luY1Jlc3BvbnNlIibC8xgiCAIQBCoMCgZvcmdfaWQQAhgBMAE6AhABQAFIA1ACWANgARKlAQoMRGVsZXRlU291cmNlEiUuc2Fhcy5hY2NvdW50cy52MS5EZWxldGVTb3VyY2VSZXF1ZXN0GiYuc2Fhcy5hY2NvdW50cy52MS5EZWxldGVTb3VyY2VSZXNwb25zZSJGwvMYQggCEAQqDAoGb3JnX2lkEAIYATABOiIKHnNhYXMuZGF0YXNvdXJjZS5zb3VyY2UucmVtb3ZlZBACQAFIBFACWAJgARLEAQoTQmVnaW5HaXRIdWJBcHBTZXR1cBIsLnNhYXMuYWNjb3VudHMudjEuQmVnaW5HaXRIdWJBcHBTZXR1cFJlcXVlc3QaLS5zYWFzLmFjY291bnRzLnYxLkJlZ2luR2l0SHViQXBwU2V0dXBSZXNwb25zZSJQwvMYTAgCEAQqDAoGb3JnX2lkEAIYATABOiwKKHNhYXMuZGF0YXNvdXJjZS5naXRodWJfYXBwLnNldHVwX3N0YXJ0ZWQQAkABSARQAlgEYAESzwEKFkNvbXBsZXRlR2l0SHViQXBwU2V0dXASLy5zYWFzLmFjY291bnRzLnYxLkNvbXBsZXRlR2l0SHViQXBwU2V0dXBSZXF1ZXN0GjAuc2Fhcy5hY2NvdW50cy52MS5Db21wbGV0ZUdpdEh1YkFwcFNldHVwUmVzcG9uc2UiUsLzGE4IAhAEKgwKBm9yZ19pZBACGAEwATouCipzYWFzLmRhdGFzb3VyY2UuZ2l0aHViX2FwcC5zZXR1cF9jb21wbGV0ZWQQAkABSARQBFgDYAESzQEKGE1pZ3JhdGVHaXRIdWJTb3VyY2VUb0FwcBIxLnNhYXMuYWNjb3VudHMudjEuTWlncmF0ZUdpdEh1YlNvdXJjZVRvQXBwUmVxdWVzdBoyLnNhYXMuYWNjb3VudHMudjEuTWlncmF0ZUdpdEh1YlNvdXJjZVRvQXBwUmVzcG9uc2UiSsLzGEYIAhAEKgwKBm9yZ19pZBACGAEwATomCiJzYWFzLmRhdGFzb3VyY2UuY3JlZGVudGlhbC51cGRhdGVkEAJAAUgEUAJYA2ABEtUBChpCZWdpbkRhdGFzb3VyY2VBY2NvdW50TGluaxIzLnNhYXMuYWNjb3VudHMudjEuQmVnaW5EYXRhc291cmNlQWNjb3VudExpbmtSZXF1ZXN0GjQuc2Fhcy5hY2NvdW50cy52MS5CZWdpbkRhdGFzb3VyY2VBY2NvdW50TGlua1Jlc3BvbnNlIkzC8xhICAIQAyoMCgZvcmdfaWQQAhgBMAE6KAokc2Fhcy5kYXRhc291cmNlLmFjY291bnRfbGlua19zdGFydGVkEAJAAUgEUAJYBGABEtgBCh1Db21wbGV0ZURhdGFzb3VyY2VBY2NvdW50TGluaxI2LnNhYXMuYWNjb3VudHMudjEuQ29tcGxldGVEYXRhc291cmNlQWNjb3VudExpbmtSZXF1ZXN0Gjcuc2Fhcy5hY2NvdW50cy52MS5Db21wbGV0ZURhdGFzb3VyY2VBY2NvdW50TGlua1Jlc3BvbnNlIkbC8xhCCAIQAyoMCgZvcmdfaWQQAhgBMAE6Igoec2Fhcy5kYXRhc291cmNlLmFjY291bnRfbGlua2VkEAJAAUgEUARYA2ABErUBChxMaXN0TXlEYXRhc291cmNlQWNjb3VudExpbmtzEjUuc2Fhcy5hY2NvdW50cy52MS5MaXN0TXlEYXRhc291cmNlQWNjb3VudExpbmtzUmVxdWVzdBo2LnNhYXMuYWNjb3VudHMudjEuTGlzdE15RGF0YXNvdXJjZUFjY291bnRMaW5rc1Jlc3BvbnNlIibC8xgiCAIQAyoMCgZvcmdfaWQQAhgBMAE6AhABQAFIA1ACWANgARLUAQobRGVsZXRlRGF0YXNvdXJjZUFjY291bnRMaW5rEjQuc2Fhcy5hY2NvdW50cy52MS5EZWxldGVEYXRhc291cmNlQWNjb3VudExpbmtSZXF1ZXN0GjUuc2Fhcy5hY2NvdW50cy52MS5EZWxldGVEYXRhc291cmNlQWNjb3VudExpbmtSZXNwb25zZSJIwvMYRAgCEAMqDAoGb3JnX2lkEAIYATABOiQKIHNhYXMuZGF0YXNvdXJjZS5hY2NvdW50X3VubGlua2VkEAJAAUgEUAJYA2ABEqMBChZHZXREYXRhc291cmNlRGlyZWN0b3J5Ei8uc2Fhcy5hY2NvdW50cy52MS5HZXREYXRhc291cmNlRGlyZWN0b3J5UmVxdWVzdBowLnNhYXMuYWNjb3VudHMudjEuR2V0RGF0YXNvdXJjZURpcmVjdG9yeVJlc3BvbnNlIibC8xgiCAIQBCoMCgZvcmdfaWQQAhgBMAE6AhABQAFIA1ACWANgARK3AQoTQmluZERhdGFzb3VyY2VHcm91cBIsLnNhYXMuYWNjb3VudHMudjEuQmluZERhdGFzb3VyY2VHcm91cFJlcXVlc3QaLS5zYWFzLmFjY291bnRzLnYxLkJpbmREYXRhc291cmNlR3JvdXBSZXNwb25zZSJDwvMYPwgCEAQqDAoGb3JnX2lkEAIYATABOh8KG3NhYXMuZGF0YXNvdXJjZS5ncm91cF9ib3VuZBACQAFIBFACWANgARK/AQoVVW5iaW5kRGF0YXNvdXJjZUdyb3VwEi4uc2Fhcy5hY2NvdW50cy52MS5VbmJpbmREYXRhc291cmNlR3JvdXBSZXF1ZXN0Gi8uc2Fhcy5hY2NvdW50cy52MS5VbmJpbmREYXRhc291cmNlR3JvdXBSZXNwb25zZSJFwvMYQQgCEAQqDAoGb3JnX2lkEAIYATABOiEKHXNhYXMuZGF0YXNvdXJjZS5ncm91cF91bmJvdW5kEAJAAUgEUAJYA2ABEsABChVDbGFpbURhdGFzb3VyY2VEb21haW4SLi5zYWFzLmFjY291bnRzLnYxLkNsYWltRGF0YXNvdXJjZURvbWFpblJlcXVlc3QaLy5zYWFzLmFjY291bnRzLnYxLkNsYWltRGF0YXNvdXJjZURvbWFpblJlc3BvbnNlIkbC8xhCCAIQBCoMCgZvcmdfaWQQAhgBMAE6Igoec2Fhcy5kYXRhc291cmNlLmRvbWFpbl9jbGFpbWVkEAJAAUgEUAJYA2ABEsQBChZWZXJpZnlEYXRhc291cmNlRG9tYWluEi8uc2Fhcy5hY2NvdW50cy52MS5WZXJpZnlEYXRhc291cmNlRG9tYWluUmVxdWVzdBowLnNhYXMuYWNjb3VudHMudjEuVmVyaWZ5RGF0YXNvdXJjZURvbWFpblJlc3BvbnNlIkfC8xhDCAIQBCoMCgZvcmdfaWQQAhgBMAE6Iwofc2Fhcy5kYXRhc291cmNlLmRvbWFpbl92ZXJpZmllZBACQAFIBFACWANgARLDAQoWRGVsZXRlRGF0YXNvdXJjZURvbWFpbhIvLnNhYXMuYWNjb3VudHMudjEuRGVsZXRlRGF0YXNvdXJjZURvbWFpblJlcXVlc3QaMC5zYWFzLmFjY291bnRzLnYxLkRlbGV0ZURhdGFzb3VyY2VEb21haW5SZXNwb25zZSJGwvMYQggCEAQqDAoGb3JnX2lkEAIYATABOiIKHnNhYXMuZGF0YXNvdXJjZS5kb21haW5fcmVtb3ZlZBACQAFIBFACWANgARKgAQoVTGlzdFNvdXJjZURlbGVnYXRpb25zEi4uc2Fhcy5hY2NvdW50cy52MS5MaXN0U291cmNlRGVsZWdhdGlvbnNSZXF1ZXN0Gi8uc2Fhcy5hY2NvdW50cy52MS5MaXN0U291cmNlRGVsZWdhdGlvbnNSZXNwb25zZSImwvMYIggCEAQqDAoGb3JnX2lkEAIYATABOgIQAUABSANQAlgDYAESxwEKFlJldm9rZVNvdXJjZURlbGVnYXRpb24SLy5zYWFzLmFjY291bnRzLnYxLlJldm9rZVNvdXJjZURlbGVnYXRpb25SZXF1ZXN0GjAuc2Fhcy5hY2NvdW50cy52MS5SZXZva2VTb3VyY2VEZWxlZ2F0aW9uUmVzcG9uc2UiSsLzGEYIAhAEKgwKBm9yZ19pZBACGAEwATomCiJzYWFzLmRhdGFzb3VyY2UuZGVsZWdhdGlvbi5yZXZva2VkEAJAAUgEUAJYA2ABEusBChZEZWxlZ2F0ZVNvdXJjZVRyaWdnZXJzEi8uc2Fhcy5hY2NvdW50cy52MS5EZWxlZ2F0ZVNvdXJjZVRyaWdnZXJzUmVxdWVzdBowLnNhYXMuYWNjb3VudHMudjEuRGVsZWdhdGVTb3VyY2VUcmlnZ2Vyc1Jlc3BvbnNlIm7C8xhqCAIQBCoMCgZvcmdfaWQQAhgBMAE6Sgoic2Fhcy5kYXRhc291cmNlLmRlbGVnYXRpb24uY3JlYXRlZAoic2Fhcy5kYXRhc291cmNlLmRlbGVnYXRpb24ucmV2b2tlZBACQAFIBFACWANgARKyAQobR2V0U291cmNlVHJpZ2dlckRlY2xhcmF0aW9uEjQuc2Fhcy5hY2NvdW50cy52MS5HZXRTb3VyY2VUcmlnZ2VyRGVjbGFyYXRpb25SZXF1ZXN0GjUuc2Fhcy5hY2NvdW50cy52MS5HZXRTb3VyY2VUcmlnZ2VyRGVjbGFyYXRpb25SZXNwb25zZSImwvMYIggCEAQqDAoGb3JnX2lkEAIYATABOgIQAUABSANQAlgDYAESgwMKD1JlY29uY2lsZVNvdXJjZRIoLnNhYXMuYWNjb3VudHMudjEuUmVjb25jaWxlU291cmNlUmVxdWVzdBopLnNhYXMuYWNjb3VudHMudjEuUmVjb25jaWxlU291cmNlUmVzcG9uc2UimgLC8xhqCAIQAxoRZGF0YXNvdXJjZTppbnZva2UiEWRhdGFzb3VyY2U6aW52b2tlKg8KCXNvdXJjZV9pZBAEGAMwATohCh1zYWFzLmRhdGFzb3VyY2Uuc291cmNlLnN5bmNlZBACQANIBFACWAJgAdrzGKcBCgIIPBIDCNgEGAUiAggCKgtVTkFWQUlMQUJMRSoSUkVTT1VSQ0VfRVhIQVVTVEVEKgdBQk9SVEVEMg9zYWFzLWRhdGFzb3VyY2VKOS9zYWFzLmFjY291bnRzLnYxLkRhdGFzb3VyY2VTZXJ2aWNlL0xvb2t1cFJlY29uY2lsZVNvdXJjZVCAIFiAIGABchgKBnNvdXJjZRIGaW52b2tlEgRyZWFkGAESvwEKFUxvb2t1cFJlY29uY2lsZVNvdXJjZRIuLnNhYXMuYWNjb3VudHMudjEuTG9va3VwUmVjb25jaWxlU291cmNlUmVxdWVzdBopLnNhYXMuYWNjb3VudHMudjEuUmVjb25jaWxlU291cmNlUmVzcG9uc2UiS8LzGEcIAhADGg9kYXRhc291cmNlOnJlYWQiD2RhdGFzb3VyY2U6cmVhZCoPCgllZmZlY3RfaWQQBBgDMAE6AhABQAFIA1ACWAJgAWIGcHJvdG8z", [file_buf_validate_validate, file_codefly_runnable_v0_options, file_google_api_annotations, file_google_protobuf_duration, file_google_protobuf_struct, file_google_protobuf_timestamp, file_saas_jobs_v1_jobs, file_saas_policy_v1_options]); + fileDesc("CiFzYWFzL2FjY291bnRzL3YxL2RhdGFzb3VyY2UucHJvdG8SEHNhYXMuYWNjb3VudHMudjEiXgoWR2l0SHViRGF0YXNvdXJjZUNvbmZpZxIMCgRyZXBvGAEgASgJEg0KBXBhdGhzGAIgAygJEg4KBmJyYW5jaBgDIAEoCRIXCg9maWxlX2V4dGVuc2lvbnMYBCADKAki9AEKD0FwaU9BdXRoMkNvbmZpZxI2CgVncmFudBgEIAEoDjInLnNhYXMuYWNjb3VudHMudjEuQXBpT0F1dGgyQ29uZmlnLkdyYW50EhEKCXRva2VuX3VybBgBIAEoCRIRCgljbGllbnRfaWQYAiABKAkSDgoGc2NvcGVzGAMgAygJInMKBUdyYW50EhUKEUdSQU5UX1VOU1BFQ0lGSUVEEAASFwoTR1JBTlRfUkVGUkVTSF9UT0tFThABEhwKGEdSQU5UX0NMSUVOVF9DUkVERU5USUFMUxACEhwKGEdSQU5UX0FVVEhPUklaQVRJT05fQ09ERRADIuoBChNBcGlEYXRhc291cmNlQ29uZmlnEhAKCGJhc2VfdXJsGAEgASgJEhUKDXJlc291cmNlX3BhdGgYAiABKAkSPAoPY3JlZGVudGlhbF9raW5kGAMgASgOMiMuc2Fhcy5hY2NvdW50cy52MS5BcGlDcmVkZW50aWFsS2luZBIZChFjcmVkZW50aWFsX2hlYWRlchgEIAEoCRIeChZjcmVkZW50aWFsX3F1ZXJ5X3BhcmFtGAUgASgJEjEKBm9hdXRoMhgGIAEoCzIhLnNhYXMuYWNjb3VudHMudjEuQXBpT0F1dGgyQ29uZmlnIkEKF0NyYXdsZXJEYXRhc291cmNlQ29uZmlnEhMKC3NpdGVtYXBfdXJsGAEgASgJEhEKCW1heF9wYWdlcxgCIAEoDSKrAgoWVXBsb2FkRGF0YXNvdXJjZUNvbmZpZxIQCghlbmRwb2ludBgBIAEoCRIOCgZyZWdpb24YAiABKAkSDgoGYnVja2V0GAMgASgJEg4KBnByZWZpeBgEIAEoCRIVCg1hY2Nlc3Nfa2V5X2lkGAUgASgJEhMKC21heF9vYmplY3RzGAYgASgNEisKEGluY2x1ZGVfcGF0dGVybnMYByADKAlCEbpIDpIBCxAQIgdyBRABGIACEisKEGV4Y2x1ZGVfcGF0dGVybnMYCCADKAlCEbpIDpIBCxAQIgdyBRABGIACEkkKD2ZpbGVfZXh0ZW5zaW9ucxgJIAMoCUIwukgtkgEqECAiJnIkEAIYITIeXlwuW0EtWmEtejAtOV1bQS1aYS16MC05Ll8tXSokIoMHCgpEYXRhc291cmNlEgoKAmlkGAEgASgJEg4KBm9yZ19pZBgCIAEoCRI2Cghwcm92aWRlchgDIAEoDjIkLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZVByb3ZpZGVyEjgKBmdpdGh1YhgFIAEoCzIoLnNhYXMuYWNjb3VudHMudjEuR2l0SHViRGF0YXNvdXJjZUNvbmZpZxIyCgZzdGF0dXMYBiABKA4yIi5zYWFzLmFjY291bnRzLnYxLkRhdGFzb3VyY2VTdGF0dXMSGgoSd2ViaG9va19jb25maWd1cmVkGAcgASgIEi4KCmNyZWF0ZWRfYXQYCCABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEi4KCnVwZGF0ZWRfYXQYCSABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEjIKDmxhc3Rfc3luY2VkX2F0GAogASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIyCgNhcGkYCyABKAsyJS5zYWFzLmFjY291bnRzLnYxLkFwaURhdGFzb3VyY2VDb25maWcSOgoHY3Jhd2xlchgMIAEoCzIpLnNhYXMuYWNjb3VudHMudjEuQ3Jhd2xlckRhdGFzb3VyY2VDb25maWcSOAoGdXBsb2FkGA0gASgLMiguc2Fhcy5hY2NvdW50cy52MS5VcGxvYWREYXRhc291cmNlQ29uZmlnEhgKEGJvdW5kYXJ5X25vZGVfaWQYDiABKAkSNAoQbGFzdF9pbmdlc3RlZF9hdBgPIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASHAoUbGFzdF9pbmdlc3RlZF9jb21taXQYECABKAkSFQoNc3RhdHVzX3JlYXNvbhgRIAEoCRIWCg5ib3VuZGFyeV9sYWJlbBgSIAEoCRISCgpjb25mb3JtYW50GBMgASgIEhcKD2NvbmZvcm1hbmNlX2dhcBgUIAEoCRI/Cg1saXZlX2RlbGl2ZXJ5GBUgASgOMiguc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlTGl2ZURlbGl2ZXJ5EjUKEnJlY29uY2lsZV9pbnRlcnZhbBgWIAEoCzIZLmdvb2dsZS5wcm90b2J1Zi5EdXJhdGlvbkoECAQQBVIRdGFyZ2V0X2NvbGxlY3Rpb24isgMKFkFkZEdpdEh1YlNvdXJjZVJlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARI7CgRyZXBvGAIgASgJQi26SCpyKBADGP8BMiFeW0EtWmEtejAtOV8uLV0rL1tBLVphLXowLTlfLi1dKyQSIAoFcGF0aHMYAyADKAlCEbpIDpIBCxBAIgdyBRABGIAEEhgKBmJyYW5jaBgEIAEoCUIIukgFcgMY/wESJAoQYm91bmRhcnlfbm9kZV9pZBgIIAEoCUIIukgFcgOwAQFIABImChBjb2xsZWN0aW9uX2xhYmVsGAkgASgJQgq6SAdyBRABGP8BSAASHgoMYWNjZXNzX3Rva2VuGAYgASgJQgi6SAVyAxiACBIgCg53ZWJob29rX3NlY3JldBgHIAEoCUIIukgFcgMYgAgSSQoPZmlsZV9leHRlbnNpb25zGAogAygJQjC6SC2SASoQICImciQQAhghMh5eXC5bQS1aYS16MC05XVtBLVphLXowLTkuXy1dKiRCEQoIYm91bmRhcnkSBbpIAggBSgQIBRAGUhF0YXJnZXRfY29sbGVjdGlvbiJLChdBZGRHaXRIdWJTb3VyY2VSZXNwb25zZRIwCgpkYXRhc291cmNlGAEgASgLMhwuc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlIsgEChBBZGRTb3VyY2VSZXF1ZXN0EhgKBm9yZ19pZBgBIAEoCUIIukgFcgOwAQESQgoIcHJvdmlkZXIYAiABKA4yJC5zYWFzLmFjY291bnRzLnYxLkRhdGFzb3VyY2VQcm92aWRlckIKukgHggEEEAEgABI6CgZnaXRodWIYBCABKAsyKC5zYWFzLmFjY291bnRzLnYxLkdpdEh1YkRhdGFzb3VyY2VDb25maWdIABI0CgNhcGkYBSABKAsyJS5zYWFzLmFjY291bnRzLnYxLkFwaURhdGFzb3VyY2VDb25maWdIABI8CgdjcmF3bGVyGAggASgLMikuc2Fhcy5hY2NvdW50cy52MS5DcmF3bGVyRGF0YXNvdXJjZUNvbmZpZ0gAEjoKBnVwbG9hZBgJIAEoCzIoLnNhYXMuYWNjb3VudHMudjEuVXBsb2FkRGF0YXNvdXJjZUNvbmZpZ0gAEiQKEGJvdW5kYXJ5X25vZGVfaWQYDCABKAlCCLpIBXIDsAEBSAESJgoQY29sbGVjdGlvbl9sYWJlbBgNIAEoCUIKukgHcgUQARj/AUgBEhwKCmNyZWRlbnRpYWwYBiABKAlCCLpIBXIDGIAgEiAKDndlYmhvb2tfc2VjcmV0GAcgASgJQgi6SAVyAxiACBImChRvYXV0aDJfY2xpZW50X3NlY3JldBgKIAEoCUIIukgFcgMYgAhCCAoGY29uZmlnQhEKCGJvdW5kYXJ5EgW6SAIIAUoECAMQBFIRdGFyZ2V0X2NvbGxlY3Rpb24iRQoRQWRkU291cmNlUmVzcG9uc2USMAoKZGF0YXNvdXJjZRgBIAEoCzIcLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZSLkAgoTVXBkYXRlU291cmNlUmVxdWVzdBIYCgZvcmdfaWQYASABKAlCCLpIBXIDsAEBEhQKAmlkGAIgASgJQgi6SAVyA7ABARIYCgZicmFuY2gYAyABKAlCCLpIBXIDGP8BEiAKBXBhdGhzGAQgAygJQhG6SA6SAQsQQCIHcgUQARiABBJJCg9maWxlX2V4dGVuc2lvbnMYBSADKAlCMLpILZIBKhAgIiZyJBACGCEyHl5cLltBLVphLXowLTldW0EtWmEtejAtOS5fLV0qJBI8CghleHBlY3RlZBgGIAEoCzIiLnNhYXMuYWNjb3VudHMudjEuU291cmNlRmV0Y2hTY29wZUIGukgDyAEBEisKEGluY2x1ZGVfcGF0dGVybnMYByADKAlCEbpIDpIBCxAQIgdyBRABGIACEisKEGV4Y2x1ZGVfcGF0dGVybnMYCCADKAlCEbpIDpIBCxAQIgdyBRABGIACIs8BChBTb3VyY2VGZXRjaFNjb3BlEhgKBmJyYW5jaBgBIAEoCUIIukgFcgMY/wESIAoFcGF0aHMYAiADKAlCEbpIDpIBCxBAIgdyBRABGIAEEikKD2ZpbGVfZXh0ZW5zaW9ucxgDIAMoCUIQukgNkgEKECAiBnIEEAEYIRIpChBpbmNsdWRlX3BhdHRlcm5zGAQgAygJQg+6SAySAQkQECIFcgMYgAISKQoQZXhjbHVkZV9wYXR0ZXJucxgFIAMoCUIPukgMkgEJEBAiBXIDGIACIlgKFFVwZGF0ZVNvdXJjZVJlc3BvbnNlEjAKCmRhdGFzb3VyY2UYASABKAsyHC5zYWFzLmFjY291bnRzLnYxLkRhdGFzb3VyY2USDgoGam9iX2lkGAIgASgJIloKFURhdGFzb3VyY2VDb25maWdGaWVsZBILCgNrZXkYASABKAkSFAoMZGlzcGxheV9uYW1lGAIgASgJEgwKBGhlbHAYAyABKAkSEAoIcmVxdWlyZWQYBCABKAgixAEKGURhdGFzb3VyY2VDb25uZWN0b3JCdWRnZXQSGgoSbWF4X2l0ZW1zX3Blcl9jYWxsGAEgASgNEhoKEm1heF9ieXRlc19wZXJfY2FsbBgCIAEoAxIWCg5tYXhfaXRlbV9ieXRlcxgDIAEoAxIdChVvcGVyYXRpb25zX3Blcl93aW5kb3cYBCABKA0SFgoOd2luZG93X3NlY29uZHMYBSABKAMSIAoYYmFja2dyb3VuZF9zaGFyZV9wZXJjZW50GAYgASgNIqoGChxEYXRhc291cmNlUHJvdmlkZXJEZXNjcmlwdG9yEjYKCHByb3ZpZGVyGAEgASgOMiQuc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlUHJvdmlkZXISFAoMZGlzcGxheV9uYW1lGAIgASgJEhMKC2Rlc2NyaXB0aW9uGAMgASgJEj4KDWNvbmZpZ19maWVsZHMYBCADKAsyJy5zYWFzLmFjY291bnRzLnYxLkRhdGFzb3VyY2VDb25maWdGaWVsZBIYChBzdXBwb3J0c193ZWJob29rGAUgASgIEkcKGnN1cHBvcnRlZF9jcmVkZW50aWFsX2tpbmRzGAYgAygOMiMuc2Fhcy5hY2NvdW50cy52MS5BcGlDcmVkZW50aWFsS2luZBIRCgljb25uZWN0b3IYByABKAkSQQoJaW50ZXJmYWNlGAggASgOMi4uc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlQ29ubmVjdG9ySW50ZXJmYWNlEkQKEGNyZWRlbnRpYWxfbW9kZXMYCSADKA4yKi5zYWFzLmFjY291bnRzLnYxLkRhdGFzb3VyY2VDcmVkZW50aWFsTW9kZRI/Cg1yZWFkZXJzX21vZGVsGAogASgOMiguc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlUmVhZGVyc01vZGVsEjsKBmJ1ZGdldBgLIAEoCzIrLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZUNvbm5lY3RvckJ1ZGdldBISCgpjb25mb3JtYW50GAwgASgIEhcKD2NvbmZvcm1hbmNlX2dhcBgNIAEoCRIbChNhY2NlcHRzX25ld19zb3VyY2VzGA4gASgIEiAKGGxpdmVfZGVsaXZlcnlfY29uZmlndXJlZBgPIAEoCBIaChJhY2NlcHRzX29wZXJhdGlvbnMYECABKAgSFgoOb3BlcmF0aW9uc19nYXAYESABKAkSGwoTb2F1dGhfYXV0aG9yaXplX3VybBgSIAEoCRIXCg9vYXV0aF90b2tlbl91cmwYEyABKAkSFAoMb2F1dGhfc2NvcGVzGBQgAygJIh0KG0dldERhdGFzb3VyY2VDYXRhbG9nUmVxdWVzdCJhChxHZXREYXRhc291cmNlQ2F0YWxvZ1Jlc3BvbnNlEkEKCXByb3ZpZGVycxgBIAMoCzIuLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZVByb3ZpZGVyRGVzY3JpcHRvciIuChJMaXN0U291cmNlc1JlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABASJIChNMaXN0U291cmNlc1Jlc3BvbnNlEjEKC2RhdGFzb3VyY2VzGAEgAygLMhwuc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlIkIKEEdldFNvdXJjZVJlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARIUCgJpZBgCIAEoCUIIukgFcgOwAQEiRQoRR2V0U291cmNlUmVzcG9uc2USMAoKZGF0YXNvdXJjZRgBIAEoCzIcLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZSJ2ChFTeW5jU291cmNlUmVxdWVzdBIYCgZvcmdfaWQYASABKAlCCLpIBXIDsAEBEhQKAmlkGAIgASgJQgi6SAVyA7ABARIeCgxhY2Nlc3NfdG9rZW4YAyABKAlCCLpIBXIDGIAgEhEKCXJlY29ubmVjdBgEIAEoCCIkChJTeW5jU291cmNlUmVzcG9uc2USDgoGam9iX2lkGAEgASgJImoKFEdldFNvdXJjZVN5bmNSZXF1ZXN0EhgKBm9yZ19pZBgBIAEoCUIIukgFcgOwAQESGwoJc291cmNlX2lkGAIgASgJQgi6SAVyA7ABARIbCgZqb2JfaWQYAyABKAlCC7pICNgBAXIDsAEBIpUBChJTb3VyY2VTeW5jRGVsaXZlcnkSGAoGam9iX2lkGAEgASgJQgi6SAVyA7ABARItCgVzdGF0ZRgCIAEoDjIWLnNhYXMuam9icy52MS5Kb2JTdGF0ZUIGukgDyAEBEjYKCWV4ZWN1dGlvbhgDIAEoCzIjLnNhYXMuam9icy52MS5Kb2JFeGVjdXRpb25SZWZlcmVuY2UiiwEKEVNvdXJjZVN5bmNDaGFuZ2VzEg0KBWZpbGVzGAEgASgNEg0KBWFkZGVkGAIgASgNEhAKCG1vZGlmaWVkGAMgASgNEg8KB2RlbGV0ZWQYBCABKA0SEwoLc3BsaXRfa25vd24YBSABKAgSEAoIc25hcHNob3QYBiABKAgSDgoGY29tbWl0GAcgASgJIq0BChFTb3VyY2VTeW5jRmFpbHVyZRI5CgZyZWFzb24YASABKA4yKS5zYWFzLmFjY291bnRzLnYxLlNvdXJjZVN5bmNGYWlsdXJlUmVhc29uEgwKBGNvZGUYAiABKAkSDwoHbWVzc2FnZRgDIAEoCRIQCghyZXRyeWluZxgEIAEoCBIsCghyZXRyeV9hdBgFIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAihAQKElNvdXJjZVN5bmNQcm9ncmVzcxIwCgVwaGFzZRgBIAEoDjIhLnNhYXMuYWNjb3VudHMudjEuU291cmNlU3luY1BoYXNlEjQKB3RyaWdnZXIYAiABKA4yIy5zYWFzLmFjY291bnRzLnYxLlNvdXJjZVN5bmNUcmlnZ2VyEi0KCXF1ZXVlZF9hdBgDIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASLwoLZmV0Y2hpbmdfYXQYBCABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEi8KC2NvbXBpbGVkX2F0GAUgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIxCg1oYW5kZWRfb2ZmX2F0GAYgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIvCgtmaW5pc2hlZF9hdBgHIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASNAoHY2hhbmdlcxgIIAEoCzIjLnNhYXMuYWNjb3VudHMudjEuU291cmNlU3luY0NoYW5nZXMSNAoHZmFpbHVyZRgJIAEoCzIjLnNhYXMuYWNjb3VudHMudjEuU291cmNlU3luY0ZhaWx1cmUSDwoHYXR0ZW1wdBgKIAEoDRIUCgxtYXhfYXR0ZW1wdHMYCyABKA0i0gEKFUdldFNvdXJjZVN5bmNSZXNwb25zZRIYCgZqb2JfaWQYASABKAlCCLpIBXIDsAEBEi0KBXN0YXRlGAIgASgOMhYuc2Fhcy5qb2JzLnYxLkpvYlN0YXRlQga6SAPIAQESOAoKZGVsaXZlcmllcxgDIAMoCzIkLnNhYXMuYWNjb3VudHMudjEuU291cmNlU3luY0RlbGl2ZXJ5EjYKCHByb2dyZXNzGAQgASgLMiQuc2Fhcy5hY2NvdW50cy52MS5Tb3VyY2VTeW5jUHJvZ3Jlc3MiRQoTRGVsZXRlU291cmNlUmVxdWVzdBIYCgZvcmdfaWQYASABKAlCCLpIBXIDsAEBEhQKAmlkGAIgASgJQgi6SAVyA7ABASIWChREZWxldGVTb3VyY2VSZXNwb25zZSJWChNHaXRIdWJBcHBSZXBvc2l0b3J5EgwKBHJlcG8YASABKAkSFgoOZGVmYXVsdF9icmFuY2gYAiABKAkSGQoRYWxyZWFkeV9jb25uZWN0ZWQYAyABKAgiNgoaQmVnaW5HaXRIdWJBcHBTZXR1cFJlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABASJxChtCZWdpbkdpdEh1YkFwcFNldHVwUmVzcG9uc2USEwoLaW5zdGFsbF91cmwYASABKAkSDQoFc3RhdGUYAiABKAkSLgoKZXhwaXJlc19hdBgDIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAinAEKHUNvbXBsZXRlR2l0SHViQXBwU2V0dXBSZXF1ZXN0EhgKBm9yZ19pZBgBIAEoCUIIukgFcgOwAQESGQoFc3RhdGUYAiABKAlCCrpIB3IFEAEY/wESLAoPaW5zdGFsbGF0aW9uX2lkGAMgASgJQhO6SBByDhABGCAyCF5bMC05XSskEhgKBGNvZGUYBCABKAlCCrpIB3IFEAEY/wEidgoeQ29tcGxldGVHaXRIdWJBcHBTZXR1cFJlc3BvbnNlEhcKD2luc3RhbGxhdGlvbl9pZBgBIAEoCRI7CgxyZXBvc2l0b3JpZXMYAiADKAsyJS5zYWFzLmFjY291bnRzLnYxLkdpdEh1YkFwcFJlcG9zaXRvcnkiUQofTWlncmF0ZUdpdEh1YlNvdXJjZVRvQXBwUmVxdWVzdBIYCgZvcmdfaWQYASABKAlCCLpIBXIDsAEBEhQKAmlkGAIgASgJQgi6SAVyA7ABASJUCiBNaWdyYXRlR2l0SHViU291cmNlVG9BcHBSZXNwb25zZRIwCgpkYXRhc291cmNlGAEgASgLMhwuc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlIsQBChVEYXRhc291cmNlQWNjb3VudExpbmsSCgoCaWQYASABKAkSDgoGb3JnX2lkGAIgASgJEg8KB3VzZXJfaWQYAyABKAkSEQoJY29ubmVjdG9yGAQgASgJEhsKE3Byb3ZpZGVyX2FjY291bnRfaWQYBSABKAkSHgoWcHJvdmlkZXJfYWNjb3VudF9sb2dpbhgGIAEoCRIuCgpjcmVhdGVkX2F0GAcgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCK3AQoWRGF0YXNvdXJjZUdyb3VwQmluZGluZxIKCgJpZBgBIAEoCRIOCgZvcmdfaWQYAiABKAkSEQoJY29ubmVjdG9yGAMgASgJEhkKEXByb3ZpZGVyX2dyb3VwX2lkGAQgASgJEg8KB3RlYW1faWQYBSABKAkSEgoKY3JlYXRlZF9ieRgGIAEoCRIuCgpjcmVhdGVkX2F0GAcgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCKUAgoYRGF0YXNvdXJjZVZlcmlmaWVkRG9tYWluEgoKAmlkGAEgASgJEg4KBm9yZ19pZBgCIAEoCRIOCgZkb21haW4YAyABKAkSOAoGc3RhdHVzGAQgASgOMiguc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlRG9tYWluU3RhdHVzEhcKD3R4dF9yZWNvcmRfbmFtZRgFIAEoCRIYChB0eHRfcmVjb3JkX3ZhbHVlGAYgASgJEi8KC3ZlcmlmaWVkX2F0GAcgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIuCgpjcmVhdGVkX2F0GAggASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcCIzChdEYXRhc291cmNlRGlyZWN0b3J5VGVhbRIKCgJpZBgBIAEoCRIMCgRuYW1lGAIgASgJIqYBCiFCZWdpbkRhdGFzb3VyY2VBY2NvdW50TGlua1JlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARI0Cgljb25uZWN0b3IYAiABKAlCIbpIHnIcEAIYIDIWXlthLXpdW2EtejAtOS1dezEsMzF9JBIeCgxyZWRpcmVjdF91cmkYAyABKAlCCLpIBXIDGIAQEhEKCXNvdXJjZV9pZBgEIAEoCSJ6CiJCZWdpbkRhdGFzb3VyY2VBY2NvdW50TGlua1Jlc3BvbnNlEhUKDWF1dGhvcml6ZV91cmwYASABKAkSDQoFc3RhdGUYAiABKAkSLgoKZXhwaXJlc19hdBgDIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAidQokQ29tcGxldGVEYXRhc291cmNlQWNjb3VudExpbmtSZXF1ZXN0EhgKBm9yZ19pZBgBIAEoCUIIukgFcgOwAQESGQoFc3RhdGUYAiABKAlCCrpIB3IFEAEYgEASGAoEY29kZRgDIAEoCUIKukgHcgUQARj/ASJeCiVDb21wbGV0ZURhdGFzb3VyY2VBY2NvdW50TGlua1Jlc3BvbnNlEjUKBGxpbmsYASABKAsyJy5zYWFzLmFjY291bnRzLnYxLkRhdGFzb3VyY2VBY2NvdW50TGluayI/CiNMaXN0TXlEYXRhc291cmNlQWNjb3VudExpbmtzUmVxdWVzdBIYCgZvcmdfaWQYASABKAlCCLpIBXIDsAEBIl4KJExpc3RNeURhdGFzb3VyY2VBY2NvdW50TGlua3NSZXNwb25zZRI2CgVsaW5rcxgBIAMoCzInLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZUFjY291bnRMaW5rIlQKIkRlbGV0ZURhdGFzb3VyY2VBY2NvdW50TGlua1JlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARIUCgJpZBgCIAEoCUIIukgFcgOwAQEiJQojRGVsZXRlRGF0YXNvdXJjZUFjY291bnRMaW5rUmVzcG9uc2UiOQodR2V0RGF0YXNvdXJjZURpcmVjdG9yeVJlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABASKLAgoeR2V0RGF0YXNvdXJjZURpcmVjdG9yeVJlc3BvbnNlEjYKBWxpbmtzGAEgAygLMicuc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlQWNjb3VudExpbmsSOgoIYmluZGluZ3MYAiADKAsyKC5zYWFzLmFjY291bnRzLnYxLkRhdGFzb3VyY2VHcm91cEJpbmRpbmcSOwoHZG9tYWlucxgDIAMoCzIqLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZVZlcmlmaWVkRG9tYWluEjgKBXRlYW1zGAQgAygLMikuc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlRGlyZWN0b3J5VGVhbSKuAQoaQmluZERhdGFzb3VyY2VHcm91cFJlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARI0Cgljb25uZWN0b3IYAiABKAlCIbpIHnIcEAIYIDIWXlthLXpdW2EtejAtOS1dezEsMzF9JBIlChFwcm92aWRlcl9ncm91cF9pZBgDIAEoCUIKukgHcgUQARj/ARIZCgd0ZWFtX2lkGAQgASgJQgi6SAVyA7ABASJYChtCaW5kRGF0YXNvdXJjZUdyb3VwUmVzcG9uc2USOQoHYmluZGluZxgBIAEoCzIoLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZUdyb3VwQmluZGluZyJOChxVbmJpbmREYXRhc291cmNlR3JvdXBSZXF1ZXN0EhgKBm9yZ19pZBgBIAEoCUIIukgFcgOwAQESFAoCaWQYAiABKAlCCLpIBXIDsAEBIh8KHVVuYmluZERhdGFzb3VyY2VHcm91cFJlc3BvbnNlIlQKHENsYWltRGF0YXNvdXJjZURvbWFpblJlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARIaCgZkb21haW4YAiABKAlCCrpIB3IFEAMY/QEiWwodQ2xhaW1EYXRhc291cmNlRG9tYWluUmVzcG9uc2USOgoGZG9tYWluGAEgASgLMiouc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlVmVyaWZpZWREb21haW4iTwodVmVyaWZ5RGF0YXNvdXJjZURvbWFpblJlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARIUCgJpZBgCIAEoCUIIukgFcgOwAQEiXAoeVmVyaWZ5RGF0YXNvdXJjZURvbWFpblJlc3BvbnNlEjoKBmRvbWFpbhgBIAEoCzIqLnNhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZVZlcmlmaWVkRG9tYWluIk8KHURlbGV0ZURhdGFzb3VyY2VEb21haW5SZXF1ZXN0EhgKBm9yZ19pZBgBIAEoCUIIukgFcgOwAQESFAoCaWQYAiABKAlCCLpIBXIDsAEBIiAKHkRlbGV0ZURhdGFzb3VyY2VEb21haW5SZXNwb25zZSKeAgoQU291cmNlRGVsZWdhdGlvbhIKCgJpZBgBIAEoCRIRCglzb3VyY2VfaWQYAiABKAkSFAoMcHJpbmNpcGFsX2lkGAMgASgJEg4KBm1vZHVsZRgEIAEoCRIPCgdiaW5kaW5nGAUgASgJEi4KCmNyZWF0ZWRfYXQYBiABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEi4KCnJldm9rZWRfYXQYByABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEkAKCnJldm9jYXRpb24YCCABKA4yLC5zYWFzLmFjY291bnRzLnYxLlNvdXJjZURlbGVnYXRpb25SZXZvY2F0aW9uEhIKCnJldm9rZWRfYnkYCSABKAkigQEKHExpc3RTb3VyY2VEZWxlZ2F0aW9uc1JlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARIgCglzb3VyY2VfaWQYAiABKAlCCLpIBXIDsAEBSACIAQESFwoPaW5jbHVkZV9yZXZva2VkGAMgASgIQgwKCl9zb3VyY2VfaWQiWAodTGlzdFNvdXJjZURlbGVnYXRpb25zUmVzcG9uc2USNwoLZGVsZWdhdGlvbnMYASADKAsyIi5zYWFzLmFjY291bnRzLnYxLlNvdXJjZURlbGVnYXRpb24iTwodUmV2b2tlU291cmNlRGVsZWdhdGlvblJlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARIUCgJpZBgCIAEoCUIIukgFcgOwAQEiWAoeUmV2b2tlU291cmNlRGVsZWdhdGlvblJlc3BvbnNlEjYKCmRlbGVnYXRpb24YASABKAsyIi5zYWFzLmFjY291bnRzLnYxLlNvdXJjZURlbGVnYXRpb24iMgoPVHJpZ2dlcklkZW50aXR5Eg4KBnNvdXJjZRgBIAEoCRIPCgdiaW5kaW5nGAIgASgJIoQBChdEZWNsYXJlZFRyaWdnZXJTY2hlZHVsZRIMCgRjcm9uGAEgASgJEhAKCHRpbWV6b25lGAIgASgJEhIKCm1pc3NlZF9ydW4YAyABKAkSDwoHb3ZlcmxhcBgEIAEoCRIQCghkZWFkbGluZRgFIAEoAxISCgppbnB1dF9qc29uGAYgASgJIucCChpUcmlnZ2VyRGVsZWdhdGlvblJlZmVyZW5jZRIRCglyZWZlcmVuY2UYASABKAkSFQoNZGVsZWdhdGlvbl9pZBgCIAEoCRIOCgZtb2R1bGUYAyABKAkSDwoHYmluZGluZxgEIAEoCRIQCghhdWRpZW5jZRgFIAEoCRIOCgZ0ZW5hbnQYBiABKAkSGgoSb3duZXJfcHJpbmNpcGFsX2lkGAcgASgJEg4KBnNjb3BlcxgIIAMoCRIOCgZhY3RpdmUYCSABKAgSLgoKY3JlYXRlZF9hdBgKIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASLgoKcmV2b2tlZF9hdBgLIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASQAoKcmV2b2NhdGlvbhgMIAEoDjIsLnNhYXMuYWNjb3VudHMudjEuU291cmNlRGVsZWdhdGlvblJldm9jYXRpb24ivgEKE1RyaWdnZXJDaGlsZENvbnRleHQSFQoNZGVsZWdhdGlvbl9pZBgBIAEoCRIOCgZtb2R1bGUYAiABKAkSDwoHYmluZGluZxgDIAEoCRIQCghhdWRpZW5jZRgEIAEoCRIOCgZzY29wZXMYBSADKAkSDgoGbG9va3VwGAYgASgIEi0KCW1pbnRlZF9hdBgHIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASDgoGcmVjb3JkGAggASgJIs0CChhTb3VyY2VUcmlnZ2VyRGVjbGFyYXRpb24SDwoHcmVsZWFzZRgBIAEoCRIRCglvcGVyYXRpb24YAiABKAkSOwoIc2NoZWR1bGUYAyABKAsyKS5zYWFzLmFjY291bnRzLnYxLkRlY2xhcmVkVHJpZ2dlclNjaGVkdWxlEjMKCGlkZW50aXR5GAQgASgLMiEuc2Fhcy5hY2NvdW50cy52MS5UcmlnZ2VySWRlbnRpdHkSDgoGZGlnZXN0GAUgASgJEjYKEmxhc3Rfb2NjdXJyZW5jZV9hdBgGIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASNgoSbmV4dF9vY2N1cnJlbmNlX2F0GAcgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIbChN1bmRlY2xhcmFibGVfcmVhc29uGAggASgJIlYKHURlbGVnYXRlU291cmNlVHJpZ2dlcnNSZXF1ZXN0EhgKBm9yZ19pZBgBIAEoCUIIukgFcgOwAQESGwoJc291cmNlX2lkGAIgASgJQgi6SAVyA7ABASJjCh5EZWxlZ2F0ZVNvdXJjZVRyaWdnZXJzUmVzcG9uc2USQQoLZGVsZWdhdGlvbnMYASADKAsyLC5zYWFzLmFjY291bnRzLnYxLlRyaWdnZXJEZWxlZ2F0aW9uUmVmZXJlbmNlIlsKIkdldFNvdXJjZVRyaWdnZXJEZWNsYXJhdGlvblJlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARIbCglzb3VyY2VfaWQYAiABKAlCCLpIBXIDsAEBIuIBCiNHZXRTb3VyY2VUcmlnZ2VyRGVjbGFyYXRpb25SZXNwb25zZRI/CgtkZWNsYXJhdGlvbhgBIAEoCzIqLnNhYXMuYWNjb3VudHMudjEuU291cmNlVHJpZ2dlckRlY2xhcmF0aW9uEkEKC2RlbGVnYXRpb25zGAIgAygLMiwuc2Fhcy5hY2NvdW50cy52MS5UcmlnZ2VyRGVsZWdhdGlvblJlZmVyZW5jZRI3CghjaGlsZHJlbhgDIAMoCzIlLnNhYXMuYWNjb3VudHMudjEuVHJpZ2dlckNoaWxkQ29udGV4dCJuChZSZWNvbmNpbGVTb3VyY2VSZXF1ZXN0EhgKBm9yZ19pZBgBIAEoCUIIukgFcgOwAQESGwoJc291cmNlX2lkGAIgASgJQgi6SAVyA7ABARIdCgllZmZlY3RfaWQYAyABKAlCCrpIB3IFEAEYgAEiTAoXUmVjb25jaWxlU291cmNlUmVzcG9uc2USEQoJZWZmZWN0X2lkGAEgASgJEg4KBnN0YXR1cxgCIAEoCRIOCgZqb2JfaWQYAyABKAkiPQocTG9va3VwUmVjb25jaWxlU291cmNlUmVxdWVzdBIdCgllZmZlY3RfaWQYASABKAlCCrpIB3IFEAEYgAEi8QIKD1NvdXJjZU9wZXJhdGlvbhIMCgRuYW1lGAEgASgJEhMKC2Rlc2NyaXB0aW9uGAIgASgJEg4KBm1ldGhvZBgDIAEoCRIMCgRwYXRoGAQgASgJEg0KBXF1ZXJ5GAUgAygJEi0KDGlucHV0X3NjaGVtYRgGIAEoCzIXLmdvb2dsZS5wcm90b2J1Zi5TdHJ1Y3QSLgoNb3V0cHV0X3NjaGVtYRgHIAEoCzIXLmdvb2dsZS5wcm90b2J1Zi5TdHJ1Y3QSOAoGZWZmZWN0GAggASgOMiguc2Fhcy5hY2NvdW50cy52MS5Tb3VyY2VPcGVyYXRpb24uRWZmZWN0EhgKEG1heF9vdXRwdXRfYnl0ZXMYCSABKA0SDgoGZGlnZXN0GAogASgJIksKBkVmZmVjdBIWChJFRkZFQ1RfVU5TUEVDSUZJRUQQABIUChBFRkZFQ1RfUkVBRF9PTkxZEAESEwoPRUZGRUNUX01VVEFUSU9OEAIijgEKHkRlY2xhcmVTb3VyY2VPcGVyYXRpb25zUmVxdWVzdBIYCgZvcmdfaWQYASABKAlCCLpIBXIDsAEBEhsKCXNvdXJjZV9pZBgCIAEoCUIIukgFcgOwAQESNQoKb3BlcmF0aW9ucxgDIAMoCzIhLnNhYXMuYWNjb3VudHMudjEuU291cmNlT3BlcmF0aW9uIlgKH0RlY2xhcmVTb3VyY2VPcGVyYXRpb25zUmVzcG9uc2USNQoKb3BlcmF0aW9ucxgBIAMoCzIhLnNhYXMuYWNjb3VudHMudjEuU291cmNlT3BlcmF0aW9uIlQKG0xpc3RTb3VyY2VPcGVyYXRpb25zUmVxdWVzdBIYCgZvcmdfaWQYASABKAlCCLpIBXIDsAEBEhsKCXNvdXJjZV9pZBgCIAEoCUIIukgFcgOwAQEiVQocTGlzdFNvdXJjZU9wZXJhdGlvbnNSZXNwb25zZRI1CgpvcGVyYXRpb25zGAEgAygLMiEuc2Fhcy5hY2NvdW50cy52MS5Tb3VyY2VPcGVyYXRpb24irgIKHEludm9rZVNvdXJjZU9wZXJhdGlvblJlcXVlc3QSGAoGb3JnX2lkGAEgASgJQgi6SAVyA7ABARIbCglzb3VyY2VfaWQYAiABKAlCCLpIBXIDsAEBEjAKCW9wZXJhdGlvbhgDIAEoCUIdukgachgyFl5bYS16XVthLXowLTlfXXswLDYzfSQSHQoKaW5wdXRfanNvbhgEIAEoCUIJukgGcgQogIAEEhsKCWVmZmVjdF9pZBgFIAEoCUIIukgFcgMogAESSQobZXhwZWN0ZWRfZGVjbGFyYXRpb25fZGlnZXN0GAYgASgJQh+6SBxyGjIYXihzaGEyNTY6WzAtOWEtZl17NjR9KT8kSACIAQFCHgocX2V4cGVjdGVkX2RlY2xhcmF0aW9uX2RpZ2VzdCJ/ChZTb3VyY2VPcGVyYXRpb25SZWNlaXB0EhEKCWVmZmVjdF9pZBgBIAEoCRIUCgxjb21taXR0ZWRfYXQYAiABKAkSDgoGc3RhdHVzGAMgASgJEhcKD3Byb3ZpZGVyX3N0YXR1cxgEIAEoDRITCgtvdXRwdXRfanNvbhgFIAEoCSJvCh1JbnZva2VTb3VyY2VPcGVyYXRpb25SZXNwb25zZRITCgtvdXRwdXRfanNvbhgBIAEoCRI5CgdyZWNlaXB0GAIgASgLMiguc2Fhcy5hY2NvdW50cy52MS5Tb3VyY2VPcGVyYXRpb25SZWNlaXB0IkMKIkxvb2t1cEludm9rZVNvdXJjZU9wZXJhdGlvblJlcXVlc3QSHQoJZWZmZWN0X2lkGAEgASgJQgq6SAdyBRABKIABInkKI1BydW5lU291cmNlT3BlcmF0aW9uUmVjZWlwdHNSZXF1ZXN0EhgKBm9yZ19pZBgBIAEoCUIIukgFcgOwAQESGwoJc291cmNlX2lkGAIgASgJQgi6SAVyA7ABARIbCgllZmZlY3RfaWQYAyABKAlCCLpIBXIDKIABIlsKJFBydW5lU291cmNlT3BlcmF0aW9uUmVjZWlwdHNSZXNwb25zZRIRCgllZmZlY3RfaWQYASABKAkSEAoIcmVjZWlwdHMYAiABKAMSDgoGc3RhdHVzGAMgASgJIkoKKUxvb2t1cFBydW5lU291cmNlT3BlcmF0aW9uUmVjZWlwdHNSZXF1ZXN0Eh0KCWVmZmVjdF9pZBgBIAEoCUIKukgHcgUQASiAASq3AQoSRGF0YXNvdXJjZVByb3ZpZGVyEiMKH0RBVEFTT1VSQ0VfUFJPVklERVJfVU5TUEVDSUZJRUQQABIeChpEQVRBU09VUkNFX1BST1ZJREVSX0dJVEhVQhABEhsKF0RBVEFTT1VSQ0VfUFJPVklERVJfQVBJEAISHwobREFUQVNPVVJDRV9QUk9WSURFUl9DUkFXTEVSEAMSHgoaREFUQVNPVVJDRV9QUk9WSURFUl9VUExPQUQQBCqRAQoQRGF0YXNvdXJjZVN0YXR1cxIhCh1EQVRBU09VUkNFX1NUQVRVU19VTlNQRUNJRklFRBAAEhwKGERBVEFTT1VSQ0VfU1RBVFVTX0FDVElWRRABEhwKGERBVEFTT1VSQ0VfU1RBVFVTX1BBVVNFRBACEh4KGkRBVEFTT1VSQ0VfU1RBVFVTX0RFR1JBREVEEAMqvAEKFkRhdGFzb3VyY2VMaXZlRGVsaXZlcnkSKAokREFUQVNPVVJDRV9MSVZFX0RFTElWRVJZX1VOU1BFQ0lGSUVEEAASIQodREFUQVNPVVJDRV9MSVZFX0RFTElWRVJZX05PTkUQARIrCidEQVRBU09VUkNFX0xJVkVfREVMSVZFUllfU09VUkNFX1dFQkhPT0sQAhIoCiREQVRBU09VUkNFX0xJVkVfREVMSVZFUllfQVBQX1dFQkhPT0sQAyrWAQoRQXBpQ3JlZGVudGlhbEtpbmQSIwofQVBJX0NSRURFTlRJQUxfS0lORF9VTlNQRUNJRklFRBAAEh4KGkFQSV9DUkVERU5USUFMX0tJTkRfQkVBUkVSEAESHQoZQVBJX0NSRURFTlRJQUxfS0lORF9CQVNJQxACEh4KGkFQSV9DUkVERU5USUFMX0tJTkRfSEVBREVSEAMSHQoZQVBJX0NSRURFTlRJQUxfS0lORF9RVUVSWRAEEh4KGkFQSV9DUkVERU5USUFMX0tJTkRfT0FVVEgyEAUqpgIKHERhdGFzb3VyY2VDb25uZWN0b3JJbnRlcmZhY2USLgoqREFUQVNPVVJDRV9DT05ORUNUT1JfSU5URVJGQUNFX1VOU1BFQ0lGSUVEEAASKAokREFUQVNPVVJDRV9DT05ORUNUT1JfSU5URVJGQUNFX0ZJTEVTEAESKAokREFUQVNPVVJDRV9DT05ORUNUT1JfSU5URVJGQUNFX1BBR0VTEAISKgomREFUQVNPVVJDRV9DT05ORUNUT1JfSU5URVJGQUNFX1JFQ09SRFMQAxIrCidEQVRBU09VUkNFX0NPTk5FQ1RPUl9JTlRFUkZBQ0VfTUVTU0FHRVMQBBIpCiVEQVRBU09VUkNFX0NPTk5FQ1RPUl9JTlRFUkZBQ0VfRVZFTlRTEAUq7AEKGERhdGFzb3VyY2VDcmVkZW50aWFsTW9kZRIqCiZEQVRBU09VUkNFX0NSRURFTlRJQUxfTU9ERV9VTlNQRUNJRklFRBAAEiMKH0RBVEFTT1VSQ0VfQ1JFREVOVElBTF9NT0RFX05PTkUQARImCiJEQVRBU09VUkNFX0NSRURFTlRJQUxfTU9ERV9PUkdfQVBQEAISKQolREFUQVNPVVJDRV9DUkVERU5USUFMX01PREVfVVNFUl9PQVVUSBADEiwKKERBVEFTT1VSQ0VfQ1JFREVOVElBTF9NT0RFX1NUQVRJQ19TRUNSRVQQBCqXAQoWRGF0YXNvdXJjZVJlYWRlcnNNb2RlbBIoCiREQVRBU09VUkNFX1JFQURFUlNfTU9ERUxfVU5TUEVDSUZJRUQQABIqCiZEQVRBU09VUkNFX1JFQURFUlNfTU9ERUxfU09VUkNFX1NDT1BFRBABEicKI0RBVEFTT1VSQ0VfUkVBREVSU19NT0RFTF9UUkFOU0xBVEVEEAIq7gEKD1NvdXJjZVN5bmNQaGFzZRIhCh1TT1VSQ0VfU1lOQ19QSEFTRV9VTlNQRUNJRklFRBAAEhwKGFNPVVJDRV9TWU5DX1BIQVNFX1FVRVVFRBABEh4KGlNPVVJDRV9TWU5DX1BIQVNFX0ZFVENISU5HEAISHgoaU09VUkNFX1NZTkNfUEhBU0VfQ09NUElMRUQQAxIgChxTT1VSQ0VfU1lOQ19QSEFTRV9IQU5ERURfT0ZGEAQSGgoWU09VUkNFX1NZTkNfUEhBU0VfRE9ORRAFEhwKGFNPVVJDRV9TWU5DX1BIQVNFX0ZBSUxFRBAGKqYDChdTb3VyY2VTeW5jRmFpbHVyZVJlYXNvbhIqCiZTT1VSQ0VfU1lOQ19GQUlMVVJFX1JFQVNPTl9VTlNQRUNJRklFRBAAEisKJ1NPVVJDRV9TWU5DX0ZBSUxVUkVfUkVBU09OX1JBVEVfTElNSVRFRBABEikKJVNPVVJDRV9TWU5DX0ZBSUxVUkVfUkVBU09OX0NSRURFTlRJQUwQAhIsCihTT1VSQ0VfU1lOQ19GQUlMVVJFX1JFQVNPTl9BQ0NFU1NfREVOSUVEEAMSKAokU09VUkNFX1NZTkNfRkFJTFVSRV9SRUFTT05fTk9UX0ZPVU5EEAQSKAokU09VUkNFX1NZTkNfRkFJTFVSRV9SRUFTT05fVE9PX0xBUkdFEAUSLworU09VUkNFX1NZTkNfRkFJTFVSRV9SRUFTT05fSE9TVF9VTkFWQUlMQUJMRRAGEiQKIFNPVVJDRV9TWU5DX0ZBSUxVUkVfUkVBU09OX09USEVSEAcSLgoqU09VUkNFX1NZTkNfRkFJTFVSRV9SRUFTT05fREVMSVZFUllfRkFJTEVEEAgqwgEKEVNvdXJjZVN5bmNUcmlnZ2VyEiMKH1NPVVJDRV9TWU5DX1RSSUdHRVJfVU5TUEVDSUZJRUQQABIeChpTT1VSQ0VfU1lOQ19UUklHR0VSX01BTlVBTBABEiEKHVNPVVJDRV9TWU5DX1RSSUdHRVJfU0NIRURVTEVEEAISHwobU09VUkNFX1NZTkNfVFJJR0dFUl9XRUJIT09LEAMSJAogU09VUkNFX1NZTkNfVFJJR0dFUl9SRUNPTkZJR1VSRUQQBCqPAQoWRGF0YXNvdXJjZURvbWFpblN0YXR1cxIoCiREQVRBU09VUkNFX0RPTUFJTl9TVEFUVVNfVU5TUEVDSUZJRUQQABIkCiBEQVRBU09VUkNFX0RPTUFJTl9TVEFUVVNfUEVORElORxABEiUKIURBVEFTT1VSQ0VfRE9NQUlOX1NUQVRVU19WRVJJRklFRBACKpUDChpTb3VyY2VEZWxlZ2F0aW9uUmV2b2NhdGlvbhIsCihTT1VSQ0VfREVMRUdBVElPTl9SRVZPQ0FUSU9OX1VOU1BFQ0lGSUVEEAASKAokU09VUkNFX0RFTEVHQVRJT05fUkVWT0NBVElPTl9SRVZPS0VEEAESKQolU09VUkNFX0RFTEVHQVRJT05fUkVWT0NBVElPTl9SRVBMQUNFRBACEi8KK1NPVVJDRV9ERUxFR0FUSU9OX1JFVk9DQVRJT05fU09VUkNFX0RFTEVURUQQAxIvCitTT1VSQ0VfREVMRUdBVElPTl9SRVZPQ0FUSU9OX01FTUJFUl9SRU1PVkVEEAQSMAosU09VUkNFX0RFTEVHQVRJT05fUkVWT0NBVElPTl9QRVJNSVNTSU9OX0xPU1QQBRIuCipTT1VSQ0VfREVMRUdBVElPTl9SRVZPQ0FUSU9OX1VTRVJfSU5BQ1RJVkUQBhIwCixTT1VSQ0VfREVMRUdBVElPTl9SRVZPQ0FUSU9OX0JJTkRJTkdfQ0hBTkdFRBAHMrE5ChFEYXRhc291cmNlU2VydmljZRK5AwocUHJ1bmVTb3VyY2VPcGVyYXRpb25SZWNlaXB0cxI1LnNhYXMuYWNjb3VudHMudjEuUHJ1bmVTb3VyY2VPcGVyYXRpb25SZWNlaXB0c1JlcXVlc3QaNi5zYWFzLmFjY291bnRzLnYxLlBydW5lU291cmNlT3BlcmF0aW9uUmVjZWlwdHNSZXNwb25zZSKpAsLzGGwIAhADGhFkYXRhc291cmNlOmludm9rZSIRZGF0YXNvdXJjZTppbnZva2UqDwoJc291cmNlX2lkEAQYAzABOiMKH3NhYXMuZGF0YXNvdXJjZS5yZWNlaXB0cy5wcnVuZWQQAkADSARQA1gDYAHa8xi0AQoCCDwSAwjYBBgFIgIIAioLVU5BVkFJTEFCTEUqElJFU09VUkNFX0VYSEFVU1RFRCoHQUJPUlRFRDIPc2Fhcy1kYXRhc291cmNlSkYvc2Fhcy5hY2NvdW50cy52MS5EYXRhc291cmNlU2VydmljZS9Mb29rdXBQcnVuZVNvdXJjZU9wZXJhdGlvblJlY2VpcHRzUIAgWIAgYAFyGAoGc291cmNlEgZpbnZva2USBHJlYWQYARLmAQoiTG9va3VwUHJ1bmVTb3VyY2VPcGVyYXRpb25SZWNlaXB0cxI7LnNhYXMuYWNjb3VudHMudjEuTG9va3VwUHJ1bmVTb3VyY2VPcGVyYXRpb25SZWNlaXB0c1JlcXVlc3QaNi5zYWFzLmFjY291bnRzLnYxLlBydW5lU291cmNlT3BlcmF0aW9uUmVjZWlwdHNSZXNwb25zZSJLwvMYRwgCEAMaD2RhdGFzb3VyY2U6cmVhZCIPZGF0YXNvdXJjZTpyZWFkKg8KCWVmZmVjdF9pZBAEGAMwAToCEAFAAUgDUANYA2ABEvwDChVJbnZva2VTb3VyY2VPcGVyYXRpb24SLi5zYWFzLmFjY291bnRzLnYxLkludm9rZVNvdXJjZU9wZXJhdGlvblJlcXVlc3QaLy5zYWFzLmFjY291bnRzLnYxLkludm9rZVNvdXJjZU9wZXJhdGlvblJlc3BvbnNlIoEDwvMYbggCEAMaEWRhdGFzb3VyY2U6aW52b2tlIhFkYXRhc291cmNlOmludm9rZSoPCglzb3VyY2VfaWQQBBgDMAE6JQohc2Fhcy5kYXRhc291cmNlLm9wZXJhdGlvbi5pbnZva2VkEARAA0gEUARYBGAB2vMYrwEKAgg8EgMI2AQYBSICCAIqC1VOQVZBSUxBQkxFKhJSRVNPVVJDRV9FWEhBVVNURUQqB0FCT1JURUQyD3NhYXMtZGF0YXNvdXJjZUo/L3NhYXMuYWNjb3VudHMudjEuRGF0YXNvdXJjZVNlcnZpY2UvTG9va3VwSW52b2tlU291cmNlT3BlcmF0aW9uUICABFiAgEBgAXIYCgZzb3VyY2USBmludm9rZRIEcmVhZBgBgtPkkwJVOgEqIlAvdjEvb3JnYW5pemF0aW9ucy97b3JnX2lkfS9kYXRhc291cmNlcy97c291cmNlX2lkfS9vcGVyYXRpb25zL3tvcGVyYXRpb259Omludm9rZRKHAgobTG9va3VwSW52b2tlU291cmNlT3BlcmF0aW9uEjQuc2Fhcy5hY2NvdW50cy52MS5Mb29rdXBJbnZva2VTb3VyY2VPcGVyYXRpb25SZXF1ZXN0Gi8uc2Fhcy5hY2NvdW50cy52MS5JbnZva2VTb3VyY2VPcGVyYXRpb25SZXNwb25zZSKAAcLzGEcIAhADGg9kYXRhc291cmNlOnJlYWQiD2RhdGFzb3VyY2U6cmVhZCoPCgllZmZlY3RfaWQQBBgDMAE6AhABQAFIA1ADWARgAYLT5JMCLxItL3YxL2RhdGFzb3VyY2Utb3BlcmF0aW9uLXJlY2VpcHRzL3tlZmZlY3RfaWR9EoUCChRMaXN0U291cmNlT3BlcmF0aW9ucxItLnNhYXMuYWNjb3VudHMudjEuTGlzdFNvdXJjZU9wZXJhdGlvbnNSZXF1ZXN0Gi4uc2Fhcy5hY2NvdW50cy52MS5MaXN0U291cmNlT3BlcmF0aW9uc1Jlc3BvbnNlIo0BwvMYRAgCEAMaD2RhdGFzb3VyY2U6cmVhZCIPZGF0YXNvdXJjZTpyZWFkKgwKBm9yZ19pZBACGAEwAToCEAFAAUgDUANYA2ABgtPkkwI/Ej0vdjEvb3JnYW5pemF0aW9ucy97b3JnX2lkfS9kYXRhc291cmNlcy97c291cmNlX2lkfS9vcGVyYXRpb25zEpMCChdEZWNsYXJlU291cmNlT3BlcmF0aW9ucxIwLnNhYXMuYWNjb3VudHMudjEuRGVjbGFyZVNvdXJjZU9wZXJhdGlvbnNSZXF1ZXN0GjEuc2Fhcy5hY2NvdW50cy52MS5EZWNsYXJlU291cmNlT3BlcmF0aW9uc1Jlc3BvbnNlIpIBwvMYRggCEAQqDAoGb3JnX2lkEAIYATABOiYKInNhYXMuZGF0YXNvdXJjZS5vcGVyYXRpb24uZGVjbGFyZWQQAkABSARQA1gDYAGC0+STAkI6ASoiPS92MS9vcmdhbml6YXRpb25zL3tvcmdfaWR9L2RhdGFzb3VyY2VzL3tzb3VyY2VfaWR9L29wZXJhdGlvbnMSrAEKD0FkZEdpdEh1YlNvdXJjZRIoLnNhYXMuYWNjb3VudHMudjEuQWRkR2l0SHViU291cmNlUmVxdWVzdBopLnNhYXMuYWNjb3VudHMudjEuQWRkR2l0SHViU291cmNlUmVzcG9uc2UiRMLzGEAIAhAEKgwKBm9yZ19pZBACGAEwATogChxzYWFzLmRhdGFzb3VyY2Uuc291cmNlLmFkZGVkEAJAAUgEUARYA2ABEpoBCglBZGRTb3VyY2USIi5zYWFzLmFjY291bnRzLnYxLkFkZFNvdXJjZVJlcXVlc3QaIy5zYWFzLmFjY291bnRzLnYxLkFkZFNvdXJjZVJlc3BvbnNlIkTC8xhACAIQBCoMCgZvcmdfaWQQAhgBMAE6IAocc2Fhcy5kYXRhc291cmNlLnNvdXJjZS5hZGRlZBACQAFIBFAEWANgARKlAQoMVXBkYXRlU291cmNlEiUuc2Fhcy5hY2NvdW50cy52MS5VcGRhdGVTb3VyY2VSZXF1ZXN0GiYuc2Fhcy5hY2NvdW50cy52MS5VcGRhdGVTb3VyY2VSZXNwb25zZSJGwvMYQggCEAQqDAoGb3JnX2lkEAIYATABOiIKHnNhYXMuZGF0YXNvdXJjZS5zb3VyY2UudXBkYXRlZBACQAFIBFACWANgARKPAQoUR2V0RGF0YXNvdXJjZUNhdGFsb2cSLS5zYWFzLmFjY291bnRzLnYxLkdldERhdGFzb3VyY2VDYXRhbG9nUmVxdWVzdBouLnNhYXMuYWNjb3VudHMudjEuR2V0RGF0YXNvdXJjZUNhdGFsb2dSZXNwb25zZSIYwvMYFAgCEAEwAToCEAFAAUgDUAJYAmABEoIBCgtMaXN0U291cmNlcxIkLnNhYXMuYWNjb3VudHMudjEuTGlzdFNvdXJjZXNSZXF1ZXN0GiUuc2Fhcy5hY2NvdW50cy52MS5MaXN0U291cmNlc1Jlc3BvbnNlIibC8xgiCAIQAyoMCgZvcmdfaWQQAhgBMAE6AhABQAFIA1ACWANgARJ8CglHZXRTb3VyY2USIi5zYWFzLmFjY291bnRzLnYxLkdldFNvdXJjZVJlcXVlc3QaIy5zYWFzLmFjY291bnRzLnYxLkdldFNvdXJjZVJlc3BvbnNlIibC8xgiCAIQAyoMCgZvcmdfaWQQAhgBMAE6AhABQAFIA1ACWANgARKeAQoKU3luY1NvdXJjZRIjLnNhYXMuYWNjb3VudHMudjEuU3luY1NvdXJjZVJlcXVlc3QaJC5zYWFzLmFjY291bnRzLnYxLlN5bmNTb3VyY2VSZXNwb25zZSJFwvMYQQgCEAQqDAoGb3JnX2lkEAIYATABOiEKHXNhYXMuZGF0YXNvdXJjZS5zb3VyY2Uuc3luY2VkEAJAAUgEUARYAmABEogBCg1HZXRTb3VyY2VTeW5jEiYuc2Fhcy5hY2NvdW50cy52MS5HZXRTb3VyY2VTeW5jUmVxdWVzdBonLnNhYXMuYWNjb3VudHMudjEuR2V0U291cmNlU3luY1Jlc3BvbnNlIibC8xgiCAIQBCoMCgZvcmdfaWQQAhgBMAE6AhABQAFIA1ACWANgARKlAQoMRGVsZXRlU291cmNlEiUuc2Fhcy5hY2NvdW50cy52MS5EZWxldGVTb3VyY2VSZXF1ZXN0GiYuc2Fhcy5hY2NvdW50cy52MS5EZWxldGVTb3VyY2VSZXNwb25zZSJGwvMYQggCEAQqDAoGb3JnX2lkEAIYATABOiIKHnNhYXMuZGF0YXNvdXJjZS5zb3VyY2UucmVtb3ZlZBACQAFIBFACWAJgARLEAQoTQmVnaW5HaXRIdWJBcHBTZXR1cBIsLnNhYXMuYWNjb3VudHMudjEuQmVnaW5HaXRIdWJBcHBTZXR1cFJlcXVlc3QaLS5zYWFzLmFjY291bnRzLnYxLkJlZ2luR2l0SHViQXBwU2V0dXBSZXNwb25zZSJQwvMYTAgCEAQqDAoGb3JnX2lkEAIYATABOiwKKHNhYXMuZGF0YXNvdXJjZS5naXRodWJfYXBwLnNldHVwX3N0YXJ0ZWQQAkABSARQAlgEYAESzwEKFkNvbXBsZXRlR2l0SHViQXBwU2V0dXASLy5zYWFzLmFjY291bnRzLnYxLkNvbXBsZXRlR2l0SHViQXBwU2V0dXBSZXF1ZXN0GjAuc2Fhcy5hY2NvdW50cy52MS5Db21wbGV0ZUdpdEh1YkFwcFNldHVwUmVzcG9uc2UiUsLzGE4IAhAEKgwKBm9yZ19pZBACGAEwATouCipzYWFzLmRhdGFzb3VyY2UuZ2l0aHViX2FwcC5zZXR1cF9jb21wbGV0ZWQQAkABSARQBFgDYAESzQEKGE1pZ3JhdGVHaXRIdWJTb3VyY2VUb0FwcBIxLnNhYXMuYWNjb3VudHMudjEuTWlncmF0ZUdpdEh1YlNvdXJjZVRvQXBwUmVxdWVzdBoyLnNhYXMuYWNjb3VudHMudjEuTWlncmF0ZUdpdEh1YlNvdXJjZVRvQXBwUmVzcG9uc2UiSsLzGEYIAhAEKgwKBm9yZ19pZBACGAEwATomCiJzYWFzLmRhdGFzb3VyY2UuY3JlZGVudGlhbC51cGRhdGVkEAJAAUgEUAJYA2ABEtUBChpCZWdpbkRhdGFzb3VyY2VBY2NvdW50TGluaxIzLnNhYXMuYWNjb3VudHMudjEuQmVnaW5EYXRhc291cmNlQWNjb3VudExpbmtSZXF1ZXN0GjQuc2Fhcy5hY2NvdW50cy52MS5CZWdpbkRhdGFzb3VyY2VBY2NvdW50TGlua1Jlc3BvbnNlIkzC8xhICAIQAyoMCgZvcmdfaWQQAhgBMAE6KAokc2Fhcy5kYXRhc291cmNlLmFjY291bnRfbGlua19zdGFydGVkEAJAAUgEUAJYBGABEtgBCh1Db21wbGV0ZURhdGFzb3VyY2VBY2NvdW50TGluaxI2LnNhYXMuYWNjb3VudHMudjEuQ29tcGxldGVEYXRhc291cmNlQWNjb3VudExpbmtSZXF1ZXN0Gjcuc2Fhcy5hY2NvdW50cy52MS5Db21wbGV0ZURhdGFzb3VyY2VBY2NvdW50TGlua1Jlc3BvbnNlIkbC8xhCCAIQAyoMCgZvcmdfaWQQAhgBMAE6Igoec2Fhcy5kYXRhc291cmNlLmFjY291bnRfbGlua2VkEAJAAUgEUARYA2ABErUBChxMaXN0TXlEYXRhc291cmNlQWNjb3VudExpbmtzEjUuc2Fhcy5hY2NvdW50cy52MS5MaXN0TXlEYXRhc291cmNlQWNjb3VudExpbmtzUmVxdWVzdBo2LnNhYXMuYWNjb3VudHMudjEuTGlzdE15RGF0YXNvdXJjZUFjY291bnRMaW5rc1Jlc3BvbnNlIibC8xgiCAIQAyoMCgZvcmdfaWQQAhgBMAE6AhABQAFIA1ACWANgARLUAQobRGVsZXRlRGF0YXNvdXJjZUFjY291bnRMaW5rEjQuc2Fhcy5hY2NvdW50cy52MS5EZWxldGVEYXRhc291cmNlQWNjb3VudExpbmtSZXF1ZXN0GjUuc2Fhcy5hY2NvdW50cy52MS5EZWxldGVEYXRhc291cmNlQWNjb3VudExpbmtSZXNwb25zZSJIwvMYRAgCEAMqDAoGb3JnX2lkEAIYATABOiQKIHNhYXMuZGF0YXNvdXJjZS5hY2NvdW50X3VubGlua2VkEAJAAUgEUAJYA2ABEqMBChZHZXREYXRhc291cmNlRGlyZWN0b3J5Ei8uc2Fhcy5hY2NvdW50cy52MS5HZXREYXRhc291cmNlRGlyZWN0b3J5UmVxdWVzdBowLnNhYXMuYWNjb3VudHMudjEuR2V0RGF0YXNvdXJjZURpcmVjdG9yeVJlc3BvbnNlIibC8xgiCAIQBCoMCgZvcmdfaWQQAhgBMAE6AhABQAFIA1ACWANgARK3AQoTQmluZERhdGFzb3VyY2VHcm91cBIsLnNhYXMuYWNjb3VudHMudjEuQmluZERhdGFzb3VyY2VHcm91cFJlcXVlc3QaLS5zYWFzLmFjY291bnRzLnYxLkJpbmREYXRhc291cmNlR3JvdXBSZXNwb25zZSJDwvMYPwgCEAQqDAoGb3JnX2lkEAIYATABOh8KG3NhYXMuZGF0YXNvdXJjZS5ncm91cF9ib3VuZBACQAFIBFACWANgARK/AQoVVW5iaW5kRGF0YXNvdXJjZUdyb3VwEi4uc2Fhcy5hY2NvdW50cy52MS5VbmJpbmREYXRhc291cmNlR3JvdXBSZXF1ZXN0Gi8uc2Fhcy5hY2NvdW50cy52MS5VbmJpbmREYXRhc291cmNlR3JvdXBSZXNwb25zZSJFwvMYQQgCEAQqDAoGb3JnX2lkEAIYATABOiEKHXNhYXMuZGF0YXNvdXJjZS5ncm91cF91bmJvdW5kEAJAAUgEUAJYA2ABEsABChVDbGFpbURhdGFzb3VyY2VEb21haW4SLi5zYWFzLmFjY291bnRzLnYxLkNsYWltRGF0YXNvdXJjZURvbWFpblJlcXVlc3QaLy5zYWFzLmFjY291bnRzLnYxLkNsYWltRGF0YXNvdXJjZURvbWFpblJlc3BvbnNlIkbC8xhCCAIQBCoMCgZvcmdfaWQQAhgBMAE6Igoec2Fhcy5kYXRhc291cmNlLmRvbWFpbl9jbGFpbWVkEAJAAUgEUAJYA2ABEsQBChZWZXJpZnlEYXRhc291cmNlRG9tYWluEi8uc2Fhcy5hY2NvdW50cy52MS5WZXJpZnlEYXRhc291cmNlRG9tYWluUmVxdWVzdBowLnNhYXMuYWNjb3VudHMudjEuVmVyaWZ5RGF0YXNvdXJjZURvbWFpblJlc3BvbnNlIkfC8xhDCAIQBCoMCgZvcmdfaWQQAhgBMAE6Iwofc2Fhcy5kYXRhc291cmNlLmRvbWFpbl92ZXJpZmllZBACQAFIBFACWANgARLDAQoWRGVsZXRlRGF0YXNvdXJjZURvbWFpbhIvLnNhYXMuYWNjb3VudHMudjEuRGVsZXRlRGF0YXNvdXJjZURvbWFpblJlcXVlc3QaMC5zYWFzLmFjY291bnRzLnYxLkRlbGV0ZURhdGFzb3VyY2VEb21haW5SZXNwb25zZSJGwvMYQggCEAQqDAoGb3JnX2lkEAIYATABOiIKHnNhYXMuZGF0YXNvdXJjZS5kb21haW5fcmVtb3ZlZBACQAFIBFACWANgARKgAQoVTGlzdFNvdXJjZURlbGVnYXRpb25zEi4uc2Fhcy5hY2NvdW50cy52MS5MaXN0U291cmNlRGVsZWdhdGlvbnNSZXF1ZXN0Gi8uc2Fhcy5hY2NvdW50cy52MS5MaXN0U291cmNlRGVsZWdhdGlvbnNSZXNwb25zZSImwvMYIggCEAQqDAoGb3JnX2lkEAIYATABOgIQAUABSANQAlgDYAESxwEKFlJldm9rZVNvdXJjZURlbGVnYXRpb24SLy5zYWFzLmFjY291bnRzLnYxLlJldm9rZVNvdXJjZURlbGVnYXRpb25SZXF1ZXN0GjAuc2Fhcy5hY2NvdW50cy52MS5SZXZva2VTb3VyY2VEZWxlZ2F0aW9uUmVzcG9uc2UiSsLzGEYIAhAEKgwKBm9yZ19pZBACGAEwATomCiJzYWFzLmRhdGFzb3VyY2UuZGVsZWdhdGlvbi5yZXZva2VkEAJAAUgEUAJYA2ABEusBChZEZWxlZ2F0ZVNvdXJjZVRyaWdnZXJzEi8uc2Fhcy5hY2NvdW50cy52MS5EZWxlZ2F0ZVNvdXJjZVRyaWdnZXJzUmVxdWVzdBowLnNhYXMuYWNjb3VudHMudjEuRGVsZWdhdGVTb3VyY2VUcmlnZ2Vyc1Jlc3BvbnNlIm7C8xhqCAIQBCoMCgZvcmdfaWQQAhgBMAE6Sgoic2Fhcy5kYXRhc291cmNlLmRlbGVnYXRpb24uY3JlYXRlZAoic2Fhcy5kYXRhc291cmNlLmRlbGVnYXRpb24ucmV2b2tlZBACQAFIBFACWANgARKyAQobR2V0U291cmNlVHJpZ2dlckRlY2xhcmF0aW9uEjQuc2Fhcy5hY2NvdW50cy52MS5HZXRTb3VyY2VUcmlnZ2VyRGVjbGFyYXRpb25SZXF1ZXN0GjUuc2Fhcy5hY2NvdW50cy52MS5HZXRTb3VyY2VUcmlnZ2VyRGVjbGFyYXRpb25SZXNwb25zZSImwvMYIggCEAQqDAoGb3JnX2lkEAIYATABOgIQAUABSANQAlgDYAESgwMKD1JlY29uY2lsZVNvdXJjZRIoLnNhYXMuYWNjb3VudHMudjEuUmVjb25jaWxlU291cmNlUmVxdWVzdBopLnNhYXMuYWNjb3VudHMudjEuUmVjb25jaWxlU291cmNlUmVzcG9uc2UimgLC8xhqCAIQAxoRZGF0YXNvdXJjZTppbnZva2UiEWRhdGFzb3VyY2U6aW52b2tlKg8KCXNvdXJjZV9pZBAEGAMwATohCh1zYWFzLmRhdGFzb3VyY2Uuc291cmNlLnN5bmNlZBACQANIBFACWAJgAdrzGKcBCgIIPBIDCNgEGAUiAggCKgtVTkFWQUlMQUJMRSoSUkVTT1VSQ0VfRVhIQVVTVEVEKgdBQk9SVEVEMg9zYWFzLWRhdGFzb3VyY2VKOS9zYWFzLmFjY291bnRzLnYxLkRhdGFzb3VyY2VTZXJ2aWNlL0xvb2t1cFJlY29uY2lsZVNvdXJjZVCAIFiAIGABchgKBnNvdXJjZRIGaW52b2tlEgRyZWFkGAESvwEKFUxvb2t1cFJlY29uY2lsZVNvdXJjZRIuLnNhYXMuYWNjb3VudHMudjEuTG9va3VwUmVjb25jaWxlU291cmNlUmVxdWVzdBopLnNhYXMuYWNjb3VudHMudjEuUmVjb25jaWxlU291cmNlUmVzcG9uc2UiS8LzGEcIAhADGg9kYXRhc291cmNlOnJlYWQiD2RhdGFzb3VyY2U6cmVhZCoPCgllZmZlY3RfaWQQBBgDMAE6AhABQAFIA1ACWAJgAWIGcHJvdG8z", [file_buf_validate_validate, file_codefly_runnable_v0_options, file_google_api_annotations, file_google_protobuf_duration, file_google_protobuf_struct, file_google_protobuf_timestamp, file_saas_jobs_v1_jobs, file_saas_policy_v1_options]); /** * GitHubDatasourceConfig is the public, non-secret configuration of a GitHub @@ -281,10 +281,40 @@ export type UploadDatasourceConfig = Message<"saas.accounts.v1.UploadDatasourceC /** * Upper bound on objects fetched per sync. Zero applies the connector default. + * Only objects that pass the filters below count against it. * * @generated from field: uint32 max_objects = 6; */ maxObjects: number; + + /** + * Which keys under the prefix are ingested. Each pattern is an RE2 regular + * expression matched against the object key with the prefix removed, + * unanchored and case-sensitive (anchor with ^ and $, fold case with (?i)). + * An empty include admits every key; an exclude match always wins. Applied + * when the prefix is listed, before anything is fetched, so an excluded + * object is never read and never counts against max_objects. An invalid + * expression is refused when the source is added or updated, never at sync: + * the call fails INVALID_ARGUMENT with a google.rpc.BadRequest violation, and + * a message that starts with the field's path, e.g. + * "upload.include_patterns[1]: …". + * + * @generated from field: repeated string include_patterns = 7; + */ + includePatterns: string[]; + + /** + * @generated from field: repeated string exclude_patterns = 8; + */ + excludePatterns: string[]; + + /** + * Case-insensitive key suffixes, e.g. [".md", ".pdf"], intersected with the + * patterns. Empty admits every type. GitHub's file_extensions semantics. + * + * @generated from field: repeated string file_extensions = 9; + */ + fileExtensions: string[]; }; /** @@ -771,6 +801,20 @@ export type UpdateSourceRequest = Message<"saas.accounts.v1.UpdateSourceRequest" * @generated from field: saas.accounts.v1.SourceFetchScope expected = 6; */ expected?: SourceFetchScope; + + /** + * An object-storage source's key patterns (see UploadDatasourceConfig), + * replaced whole like every field here. Its file_extensions is field 5; its + * branch and paths must be empty. Refused for a github source. + * + * @generated from field: repeated string include_patterns = 7; + */ + includePatterns: string[]; + + /** + * @generated from field: repeated string exclude_patterns = 8; + */ + excludePatterns: string[]; }; /** @@ -808,6 +852,16 @@ export type SourceFetchScope = Message<"saas.accounts.v1.SourceFetchScope"> & { * @generated from field: repeated string file_extensions = 3; */ fileExtensions: string[]; + + /** + * @generated from field: repeated string include_patterns = 4; + */ + includePatterns: string[]; + + /** + * @generated from field: repeated string exclude_patterns = 5; + */ + excludePatterns: string[]; }; /** diff --git a/module/services/frontend/code/packages/saas-sdk/package.json b/module/services/frontend/code/packages/saas-sdk/package.json index 3ff1eeccc..1a723c22a 100644 --- a/module/services/frontend/code/packages/saas-sdk/package.json +++ b/module/services/frontend/code/packages/saas-sdk/package.json @@ -1,6 +1,6 @@ { "name": "@codefly-dev/saas-sdk", - "version": "0.3.39", + "version": "0.3.40", "description": "Typed Connect client, gateway-bound facades, and audit data-graph tooling for the saas-starter public API.", "repository": { "type": "git", diff --git a/module/services/frontend/code/packages/saas-ui/README.md b/module/services/frontend/code/packages/saas-ui/README.md index 583ae6510..5564c19e8 100644 --- a/module/services/frontend/code/packages/saas-ui/README.md +++ b/module/services/frontend/code/packages/saas-ui/README.md @@ -417,9 +417,19 @@ refreshes sources, collection grants and readable boundaries. Preview stories in its own shell — a line in a header, say — without re-deriving them from the phase names and getting the terminal cases wrong. `` is the card, and `useSourceSync(client, orgId, source)` the read. +- `` — endpoint (https only), + region, bucket, prefix, the key filter (include and exclude RE2 patterns, one + per line, over the key after the prefix; file types as on GitHub), access key + id, the secret access key the host encrypts at receipt, and the target + collection. It states that a bucket has no per-object access list, so one + source is one collection with one set of readers. A host refusal of a pattern + leads with the field it names and is shown under that field. Map its values + with `objectStorageInputFrom(orgId, values)` for the client's optional + `addObjectStorageSource`. Nothing in the kit mounts it yet: the host refuses + new object-storage sources until the provider is conformant (#1211). - Hooks over a `DatasourceClient`: `useListSources`, `useAddGitHubSource`, - `useUpdateSource`, `useSyncSource`, `useDeleteSource`, `useAccessibleScopes`, - `useMyAccountLinks`. + `useAddObjectStorageSource`, `useUpdateSource`, `useSyncSource`, + `useDeleteSource`, `useAccessibleScopes`, `useMyAccountLinks`. - ``, behind the row's **Edit**, changes what a connected source fetches: its branch and its path and file-type filters, bound by the same schema the connect form uses. It offers the connect @@ -674,8 +684,8 @@ offer a Markdown preset. The host intersects suffixes with paths for snapshots and incremental changes; empty keeps all file types. The filter is selected at connection time and does not rewrite existing sources. -The SDK peer floor is **0.3.35**: the datasource client maps an `api` source's -credential kind and OAuth grant, and declares operations, with enums the SDK -exports from that release (principal-name resolution needs 0.3.14's directory -service). Earlier SDK versions must be rejected during installation rather than +The SDK peer floor is **0.3.40**: the object-storage connect form sends an +`upload` source's include and exclude key patterns, fields the SDK carries from +that release (the `api` source's credential kind, OAuth grant and operations +need 0.3.35; principal-name resolution needs 0.3.14's directory service). Earlier SDK versions must be rejected during installation rather than accepted and then failing when the component loads. diff --git a/module/services/frontend/code/packages/saas-ui/package.json b/module/services/frontend/code/packages/saas-ui/package.json index 775297f1b..810e70e70 100644 --- a/module/services/frontend/code/packages/saas-ui/package.json +++ b/module/services/frontend/code/packages/saas-ui/package.json @@ -1,6 +1,6 @@ { "name": "@codefly-dev/saas-ui", - "version": "0.13.2", + "version": "0.13.3", "description": "Reusable SaaS-domain frontend components (datasources, …) the portal and solutions both import, built on @codefly-dev/ui primitives and driven by an injected typed client.", "repository": { "type": "git", @@ -41,7 +41,7 @@ }, "peerDependencies": { "@bufbuild/protobuf": ">=2", - "@codefly-dev/saas-sdk": "^0.3.35", + "@codefly-dev/saas-sdk": "^0.3.40", "@connectrpc/connect": ">=2", "@connectrpc/connect-web": ">=2", "@hookform/resolvers": ">=5.1.0", @@ -53,7 +53,7 @@ }, "devDependencies": { "@bufbuild/protobuf": "^2.11.0", - "@codefly-dev/saas-sdk": "^0.3.35", + "@codefly-dev/saas-sdk": "^0.3.40", "@connectrpc/connect": "^2.1.1", "@connectrpc/connect-web": "^2.1.1", "@hookform/resolvers": "^5.9.2", diff --git a/module/services/frontend/code/packages/saas-ui/src/datasources/__tests__/connect-object-storage-form.test.tsx b/module/services/frontend/code/packages/saas-ui/src/datasources/__tests__/connect-object-storage-form.test.tsx new file mode 100644 index 000000000..2c43df225 --- /dev/null +++ b/module/services/frontend/code/packages/saas-ui/src/datasources/__tests__/connect-object-storage-form.test.tsx @@ -0,0 +1,103 @@ +import { + cleanup, + fireEvent, + render, + screen, + waitFor, +} from "@testing-library/react"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { + ConnectObjectStorageForm, + objectStorageInputFrom, +} from "../connect-object-storage-form.js"; + +afterEach(cleanup); + +function fill(label: string, value: string) { + fireEvent.change(screen.getByLabelText(label), { target: { value } }); +} + +function fillRequired() { + fill("Endpoint", "https://store.example.com"); + fill("Region", "us-east-1"); + fill("Bucket", "acme-docs"); + fill("Access key ID", "AKIAEXAMPLE"); + fill("Secret access key", "example-secret"); + fill("Target collection", "Handbook"); +} + +describe("ConnectObjectStorageForm", () => { + it("takes key patterns and file types, one pattern per line", async () => { + const onSubmit = vi.fn(); + render( + {}} + isPending={false} + />, + ); + fillRequired(); + fill("Prefix (optional)", "kb/"); + // A pattern may hold a comma; only a line ends one. + fill("Include keys matching (optional)", "^guides/\n\\.md$\n^v{1,3}/"); + fill("Exclude keys matching (optional)", "^drafts/"); + fill("File types (optional)", ".MD, .pdf"); + fireEvent.click(screen.getByRole("button", { name: "Connect bucket" })); + + await waitFor(() => expect(onSubmit).toHaveBeenCalledTimes(1)); + expect(objectStorageInputFrom("org-1", onSubmit.mock.calls[0][0])).toEqual({ + orgId: "org-1", + endpoint: "https://store.example.com", + region: "us-east-1", + bucket: "acme-docs", + prefix: "kb/", + includePatterns: ["^guides/", "\\.md$", "^v{1,3}/"], + excludePatterns: ["^drafts/"], + fileExtensions: [".md", ".pdf"], + accessKeyId: "AKIAEXAMPLE", + secretAccessKey: "example-secret", + targetCollection: "Handbook", + boundaryNodeId: undefined, + }); + }); + + it("shows the host's refusal of a pattern under the field it names", () => { + const refusal = + 'upload.exclude_patterns[0]: "^(?!drafts/)" is not a valid RE2 expression: invalid or unsupported Perl syntax: `(?!`'; + render( + {}} + onCancel={() => {}} + isPending={false} + errorMessage={refusal} + />, + ); + const field = screen.getByLabelText("Exclude keys matching (optional)"); + expect(field.getAttribute("aria-invalid")).toBe("true"); + const described = document.getElementById( + field.getAttribute("aria-describedby") ?? "", + ); + expect(described?.textContent).toBe(refusal); + // Placed once, under its field: not repeated in the form's own slot. + expect(screen.getAllByText(refusal)).toHaveLength(1); + expect( + screen + .getByLabelText("Include keys matching (optional)") + .getAttribute("aria-invalid"), + ).toBe("false"); + }); + + it("keeps a refusal that names no key-filter field in the form's own slot", () => { + render( + {}} + onCancel={() => {}} + isPending={false} + errorMessage="the store refused the access key" + />, + ); + expect(screen.getByRole("alert").textContent).toBe( + "the store refused the access key", + ); + }); +}); diff --git a/module/services/frontend/code/packages/saas-ui/src/datasources/__tests__/gateway.test.tsx b/module/services/frontend/code/packages/saas-ui/src/datasources/__tests__/gateway.test.tsx index 5c3125a61..44ebefa13 100644 --- a/module/services/frontend/code/packages/saas-ui/src/datasources/__tests__/gateway.test.tsx +++ b/module/services/frontend/code/packages/saas-ui/src/datasources/__tests__/gateway.test.tsx @@ -187,6 +187,20 @@ describe("createDatasourceClient", () => { expected: { branch: "main", paths: [], fileExtensions: [] }, }), deleteSource: () => client.deleteSource("org-1", "ds-1"), + addObjectStorageSource: () => + client.addObjectStorageSource!({ + orgId: "org-1", + endpoint: "https://store.example.com", + region: "us-east-1", + bucket: "acme-docs", + prefix: "kb/", + includePatterns: ["^guides/"], + excludePatterns: [], + fileExtensions: [], + accessKeyId: "AKIAEXAMPLE", + secretAccessKey: "secret", + targetCollection: "Example", + }), // The stub answers {} to everything; an operation whose answer must // carry a record rejects on it, after the call this test counts. beginAccountLink: () => diff --git a/module/services/frontend/code/packages/saas-ui/src/datasources/collection-field.tsx b/module/services/frontend/code/packages/saas-ui/src/datasources/collection-field.tsx new file mode 100644 index 000000000..ca8a2f39b --- /dev/null +++ b/module/services/frontend/code/packages/saas-ui/src/datasources/collection-field.tsx @@ -0,0 +1,112 @@ +"use client"; + +import { Input, Label } from "@codefly-dev/ui/layout"; +import type { UseFormRegisterReturn } from "react-hook-form"; +import { fieldErrorClass as errorClass } from "./credential-mode.js"; +import type { CollectionAccessView } from "./types.js"; + +interface CollectionFieldProps { + idFor: (name: string) => string; + collections?: CollectionAccessView[]; + readableNodeIds?: string[]; + collectionError?: boolean; + boundaryNodeId?: string; + targetCollection?: string; + /** An existing collection was picked, or "" to create one. */ + onSelect: (nodeId: string, label: string) => void; + inputProps: UseFormRegisterReturn; + error?: string; +} + +/** + * Where a new source's content lands: an existing collection, or a label to + * create one under. Every connect form shares it, so who can read the result + * is told the same way whichever provider is being connected. + */ +export function CollectionField({ + idFor, + collections, + readableNodeIds, + collectionError, + boundaryNodeId, + targetCollection, + onSelect, + inputProps, + error, +}: CollectionFieldProps) { + const selected = collections?.find( + (collection) => collection.nodeId === boundaryNodeId, + ); + const named = + selected ?? + collections?.find((collection) => collection.label === targetCollection); + return ( +
+ {collections && ( + <> + + + + )} + + +

+ Connecting grants no read access to the creator or a default team. + Organization administrators explicitly grant members or teams read + access. Platform administrators read every collection without a grant. +

+ {collectionError ? ( +

Couldn’t inspect collection grants.

+ ) : named ? ( +

+ Readers:{" "} + {named.grants.map((grant) => grant.subjectLabel).join(", ") || + "No collection read grants"} + .{" "} + {readableNodeIds === undefined + ? "Your read permission is unresolved." + : readableNodeIds.includes(named.nodeId) + ? "You can read this collection." + : "You do not have read access. Ask an administrator for access."} +

+ ) : ( +

+ {collections + ? "New collections have no read grants. Your read permission must be checked after creation." + : "Collection grants are unresolved. Inspect them in the host after connecting."} +

+ )} + {error && ( +

+ {error} +

+ )} +
+ ); +} diff --git a/module/services/frontend/code/packages/saas-ui/src/datasources/connect-github-form.tsx b/module/services/frontend/code/packages/saas-ui/src/datasources/connect-github-form.tsx index 4512cd87e..532fbb856 100644 --- a/module/services/frontend/code/packages/saas-ui/src/datasources/connect-github-form.tsx +++ b/module/services/frontend/code/packages/saas-ui/src/datasources/connect-github-form.tsx @@ -4,6 +4,7 @@ import { Button, Input, Label, Textarea } from "@codefly-dev/ui/layout"; import { zodResolver } from "@hookform/resolvers/zod"; import { useId } from "react"; import { useForm, useWatch } from "react-hook-form"; +import { CollectionField } from "./collection-field.js"; import { AccessTokenField, AppInstallPrompt, @@ -78,12 +79,6 @@ export function ConnectGitHubForm({ control: form.control, name: "targetCollection", }); - const selected = collections?.find( - (collection) => collection.nodeId === boundaryNodeId, - ); - const named = - selected ?? - collections?.find((collection) => collection.label === targetCollection); return (
-
- {collections && ( - <> - - - - )} - - -

- Connecting grants no read access to the creator or a default team. - Organization administrators explicitly grant members or teams read - access. Platform administrators read every collection without a - grant. -

- {collectionError ? ( -

Couldn’t inspect collection grants.

- ) : named ? ( -

- Readers:{" "} - {named.grants.map((grant) => grant.subjectLabel).join(", ") || - "No collection read grants"} - .{" "} - {readableNodeIds === undefined - ? "Your read permission is unresolved." - : readableNodeIds.includes(named.nodeId) - ? "You can read this collection." - : "You do not have read access. Ask an administrator for access."} -

- ) : ( -

- {collections - ? "New collections have no read grants. Your read permission must be checked after creation." - : "Collection grants are unresolved. Inspect them in the host after connecting."} -

- )} - {errors.targetCollection && ( -

- {errors.targetCollection.message} -

- )} -
+ { + form.setValue("boundaryNodeId", nodeId); + form.setValue("targetCollection", label); + }} + inputProps={form.register("targetCollection")} + {...(errors.targetCollection?.message + ? { error: errors.targetCollection.message } + : {})} + /> {method === "pat" && ( + value.toLowerCase(), + ), + accessKeyId: values.accessKeyId, + secretAccessKey: values.secretAccessKey, + targetCollection: values.targetCollection, + boundaryNodeId: values.boundaryNodeId || undefined, + }; +} + +interface ConnectObjectStorageFormProps { + collections?: CollectionAccessView[]; + readableNodeIds?: string[]; + collectionError?: boolean; + onSubmit: (values: ConnectObjectStorageValues) => void; + onCancel: () => void; + isPending: boolean; + errorMessage?: string; +} + +const textFields = [ + { + name: "endpoint", + label: "Endpoint", + placeholder: "https://s3.us-east-1.amazonaws.com", + }, + { name: "region", label: "Region", placeholder: "us-east-1" }, + { name: "bucket", label: "Bucket", placeholder: "acme-knowledge" }, + { name: "prefix", label: "Prefix (optional)", placeholder: "handbook/" }, + { name: "accessKeyId", label: "Access key ID", placeholder: "" }, +] as const; + +export function ConnectObjectStorageForm({ + collections, + readableNodeIds, + collectionError, + onSubmit, + onCancel, + isPending, + errorMessage, +}: ConnectObjectStorageFormProps) { + const fieldId = useId(); + const idFor = (name: string) => `${fieldId}-${name}`; + const form = useForm({ + resolver: zodResolver(connectObjectStorageSchema), + defaultValues: { + endpoint: "", + region: "", + bucket: "", + prefix: "", + includePatterns: "", + excludePatterns: "", + fileExtensions: "", + accessKeyId: "", + secretAccessKey: "", + boundaryNodeId: "", + targetCollection: "", + }, + }); + const { errors } = form.formState; + // The host refuses an invalid pattern by naming its field; that refusal is + // shown under the field, and only an unplaced one in the form's own slot. + const refusedField = refusedKeyFilterField(errorMessage); + const filterError = ( + name: "includePatterns" | "excludePatterns" | "fileExtensions", + ) => + errors[name]?.message ?? (refusedField === name ? errorMessage : undefined); + const filters = [ + { + name: "includePatterns", + label: "Include keys matching (optional)", + help: "One RE2 pattern per line, matched against the key after the prefix, e.g. ^guides/ or \\.md$. Empty includes every key.", + }, + { + name: "excludePatterns", + label: "Exclude keys matching (optional)", + help: "One RE2 pattern per line. An excluded key is never fetched, even if it is also included.", + }, + ] as const; + const boundaryNodeId = useWatch({ + control: form.control, + name: "boundaryNodeId", + }); + const targetCollection = useWatch({ + control: form.control, + name: "targetCollection", + }); + + return ( +
+
+
+

Connect object storage

+

+ Register a bucket as a data source. The objects under its prefix + that pass its key filter are pulled on sync. Everyone who may read + the collection reads every one of them: a bucket has no per-object + access list, so the bucket, prefix and filter make one collection + with one set of readers. For another audience, connect another + source with its own filter. +

+
+ +
+ {textFields.map((field) => { + const error = errors[field.name]?.message; + return ( +
+ + + {error && ( +

+ {error} +

+ )} +
+ ); + })} + + {filters.map((field) => { + const error = filterError(field.name); + return ( +
+ +