Skip to content

refactoring: product neutral operator core - #482

Open
hossain-rayhan wants to merge 26 commits into
documentdb:mainfrom
hossain-rayhan:developer/product-neutral-core
Open

hossain-rayhan wants to merge 26 commits into
documentdb:mainfrom
hossain-rayhan:developer/product-neutral-core

Conversation

@hossain-rayhan

@hossain-rayhan hossain-rayhan commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Extract a product-neutral operator core so other PostgreSQL-extension products can reuse DocumentDB’s orchestration without duplicating its controllers.

DocumentDB remains the first implementation. Its existing CRDs, resource naming, configuration defaults, and rendered CNPG output are preserved.

Changes

• Introduce  ProductProfile ,  ClusterIntent , and typed adapters to separate product-specific configuration from shared orchestration.
• Move cluster, certificate, persistent-volume, backup, and scheduled-backup reconciliation into generic engines. Existing DocumentDB controllers delegate to them.
• Extract shared admission validation for schema/binary compatibility, rollback protection, storage resizing, restore compatibility, and ImageVolume support.
• Expose the shared functionality through  operator/src/pkg/core , allowing consumers in separate Go modules to avoid importing internal packages.
• Provide optional replication hooks and public replication-context construction for multi-region integrations.
• Make the sidecar injector’s gateway container name, port, arguments, and credential environment-variable names configurable. Reconcile contract changes on existing clusters, including gateway Service ports.
• Make monitoring resource attributes and the PostgreSQL health metric name product-configurable.
• Document the architecture and integration pattern in product-neutral-core.md.

Compatibility

• No CRD schema or existing Kubernetes API changes.
• Generated CRDs, RBAC, and DeepCopy files are unchanged.
• Seven golden scenarios preserve byte-identical DocumentDB CNPG rendering.
• Existing plugin parameter keys and operator environment-variable names remain intact.
• New gateway parameters are optional and retain DocumentDB defaults when omitted.
• DocumentDB-specific replication, promotion, and fleet-networking behavior remains behind the shared replication hooks.

Validation

  • Changed-code (patch) coverage: 94.55%, exceeding the unchanged 90% CI requirement.
  • Added focused tests for reconciler lifecycle and error paths, certificates, backups, scheduled backups, PV recovery, replication hooks, and public core helpers.
  • Operator unit tests, race tests, vet, and build pass.
  • Sidecar unit tests, race tests, vet, and build pass.
  • All 18 e2e matrix jobs pass across amd64 and arm64, including lifecycle, scaling, data, backups/restores, TLS, features, upgrades, and cluster replication.
  • ImageVolume compatibility checks pass on Kubernetes 1.33 and 1.34.

Limitations: Live Kubernetes e2e tests were not run. The pinned linter still reports the same six pre-existing deprecation findings; no new findings were introduced.

Copilot AI balanced review requested due to automatic review settings October 20, 2026 18:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Product-defined names can bypass capability validation or break shared SQL operations.

2 open issues
What changed in this PR

Extracts reusable PostgreSQL-extension orchestration behind a public core API, with DocumentDB as its first implementation.

Changes:

  • Introduces product profiles, typed adapters, shared reconcilers, and validation helpers.
  • Makes gateway configuration and monitoring names product-configurable.
  • Adds compatibility snapshots, second-product tests, and integration documentation.
File Description
operator/​src/​pkg/​core/​example_test.go Demonstrates public rendering and replication APIs.
operator/​src/​pkg/​core/​engine.go Exposes reconcilers, adapters, and validation.
operator/​src/​pkg/​core/​engine_test.go Tests second-product reconciliation and monitoring.
operator/​src/​pkg/​core/​core.go Exposes rendering and configuration helpers.
operator/​src/​internal/​webhook/​imagevolume.go Delegates to the shared capability probe.
operator/​src/​internal/​validation/​imagevolume.go Extracts the reusable ImageVolume probe.
operator/​src/​internal/​utils/​util.go Generalizes gateway Services and reconciles ports.
operator/​src/​internal/​utils/​util_test.go Tests Service configuration and field preservation.
operator/​src/​internal/​utils/​replication_context.go Exposes replication-context construction.
operator/​src/​internal/​utils/​pv_recovery.go Supports product-specific recovery labels.
operator/​src/​internal/​utils/​constants.go Defines gateway contract parameters and defaults.
operator/​src/​internal/​product/​profile.go Adds PostgreSQL, gateway, and naming contracts.
operator/​src/​internal/​product/​intent.go Expands product-neutral rendering inputs.
operator/​src/​internal/​product/​documentdb.go Maps DocumentDB configuration into shared contracts.
operator/​src/​internal/​product/​documentdb_test.go Tests renderer-input mapping.
operator/​src/​internal/​product/​documentdb_backup.go Adds DocumentDB backup adapters.
operator/​src/​internal/​product/​cluster.go Defines cluster adapters and naming conventions.
operator/​src/​internal/​product/​cluster_test.go Tests DocumentDB contract compatibility.
operator/​src/​internal/​otel/​options_test.go Tests configurable monitoring names.
operator/​src/​internal/​otel/​config.go Renders product-specific monitoring configuration.
operator/​src/​internal/​otel/​config_test.go Adapts tests to rendered static configuration.
operator/​src/​internal/​otel/​base_config.yaml Parameterizes the health metric name.
operator/​src/​internal/​engine/​sql.go Extracts pod-based SQL execution.
operator/​src/​internal/​engine/​scheduledbackup.go Adds generic scheduled-backup reconciliation.
operator/​src/​internal/​engine/​pv_recovery.go Extracts retained-volume recovery orchestration.
operator/​src/​internal/​engine/​predicates.go Shares cluster and Service watch predicates.
operator/​src/​internal/​controller/​scheduledbackup_controller.go Delegates scheduling to the shared engine.
operator/​src/​internal/​controller/​backup_controller.go Delegates backup handling to the shared engine.
operator/​src/​internal/​cnpg/​testdata/​golden/​replica-region-ignores-recovery.json Records replica-region rendering.
operator/​src/​internal/​cnpg/​testdata/​golden/​recovery-pv.json Records retained-volume recovery rendering.
operator/​src/​internal/​cnpg/​testdata/​golden/​recovery-backup.json Records backup recovery rendering.
operator/​src/​internal/​cnpg/​testdata/​golden/​full.json Records comprehensive configuration rendering.
operator/​src/​internal/​cnpg/​testdata/​golden/​explicit-component-resources.json Records explicit resource allocation rendering.
operator/​src/​internal/​cnpg/​testdata/​golden/​env-overrides.json Records environment-override rendering.
operator/​src/​internal/​cnpg/​testdata/​golden/​defaults.json Records default DocumentDB rendering.
operator/​src/​internal/​cnpg/​second_product_test.go Tests product-neutral rendering and monitoring hashes.
operator/​src/​internal/​cnpg/​resource_validation.go Generalizes resource validation inputs and paths.
operator/​src/​internal/​cnpg/​resource_split.go Generalizes resource allocation inputs.
operator/​src/​internal/​cnpg/​resource_split_test.go Tests product-specific validation paths.
operator/​src/​internal/​cnpg/​pg_defaults.go Separates shared and product-protected parameters.
operator/​src/​internal/​cnpg/​gateway_contract_test.go Tests gateway updates, removal, and restarts.
operator/​src/​internal/​cnpg/​extension_sql.go Adds product-aware SQL and image helpers.
operator/​src/​internal/​cnpg/​extension_sql_test.go Tests extension and role helpers.
operator/​src/​internal/​cnpg/​cnpg_sync.go Synchronizes product extensions and gateway contracts.
operator/​src/​internal/​cnpg/​cnpg_sync_test.go Tests extension lookup and image synchronization.
operator/​src/​internal/​cnpg/​cnpg_intent_test.go Updates intent-only renderer calls.
operator/​src/​internal/​cnpg/​cnpg_golden_test.go Adds seven rendering compatibility scenarios.
operator/​src/​api/​preview/​scheduledbackup_funcs.go Extracts reusable scheduling helpers.
operator/​src/​api/​preview/​backup_funcs.go Extracts backup creation and status helpers.
operator/​cnpg-plugins/​sidecar-injector/​internal/​lifecycle/​lifecycle.go Applies configurable gateway container settings.
operator/​cnpg-plugins/​sidecar-injector/​internal/​lifecycle/​lifecycle_test.go Tests default and custom gateway injection.
operator/​cnpg-plugins/​sidecar-injector/​internal/​config/​config.go Parses and serializes gateway overrides.
operator/​cnpg-plugins/​sidecar-injector/​internal/​config/​config_test.go Tests gateway defaults and round trips.
docs/​developer-guides/​sidecar-injector-plugin-configuration.md Documents optional gateway parameters.
docs/​designs/​product-neutral-core.md Explains architecture and external integration.
CHANGELOG.md Summarizes the reusable core feature.
AGENTS.md Documents core boundaries and compatibility expectations.

Review effort: Balanced

Comment thread operator/src/internal/cnpg/extension_sql.go Outdated
Comment thread operator/src/internal/validation/imagevolume.go Outdated
@hossain-rayhan hossain-rayhan changed the title Developer/product neutral core refactoring: product neutral operator core Oct 20, 2026

@xgerman xgerman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The product-neutral extraction is well structured, and I did not find an additional DocumentDB behavior regression while tracing the cluster, certificate, PV, backup, and scheduled-backup paths. The focused suites also pass (go test ./internal/product ./internal/cnpg ./internal/engine ./internal/validation ./pkg/core), consistent with the green golden and E2E checks.

I am requesting changes for the two open correctness issues in the new second-product surface:

  • Product-defined extension and role names are emitted as bare SQL identifiers, so valid hyphenated or mixed-case names break extension upgrades and replication grants: #482 (comment)
  • The ImageVolume probe derives generateName from the human-readable product name. A name such as Cosmos DB makes the baseline probe invalid and the intentionally fail-open path bypasses capability enforcement: #482 (comment)

Both should be fixed with focused regression tests before merging.

hossain-rayhan and others added 22 commits October 22, 2026 11:33
Locks the rendered Cluster for a matrix of DocumentDB inputs so the
product-neutral refactor can prove byte-identical output.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
Adds product.Resources and ComputeResourceSplitFor/ValidateResourcesFor so
the resource carve-out no longer requires a DocumentDB resource. The
DocumentDB entry points delegate to them unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
Adds resources, monitoring, TLS, log level, stop delay, user parameters,the ChangeStreams gate and image pull policies to the intent and maps themfrom the DocumentDB custom resource.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
ProductProfile gains a PostgresProfile (extension, shared libraries,
bootstrap SQL, pg_hba, protected GUCs, replication admin role) and
GetCnpgClusterSpecFromIntent no longer takes a DocumentDB: every
product-varying value comes from the ClusterIntent. Golden snapshots
confirm the DocumentDB rendering is byte-identical.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
SyncCnpgCluster now identifies the product extension from the desired
cluster rather than the hardcoded documentdb name, and sidecar plugin
parameter keys are shared constants used by the renderer and the sync.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
…profile

Adds product-neutral helpers for the extension version query, ALTER
EXTENSION, the streaming_replica role grant and current image lookups.
The DocumentDB reconciler now takes the extension name and replication
admin role from its product profile; the generated SQL is unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
Adds optional gatewayContainerName, gatewayPort, gatewayArgs (JSON),
gatewayUsernameEnv and gatewayPasswordEnv plugin parameters. They
default to the DocumentDB gateway and are only serialized when they
differ, so existing clusters see no change. The operator forwards them
from ProductProfile.Gateway, which is empty for DocumentDB.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
Renders a synthetic pgcosmos intent end to end and asserts the CNPG
Cluster carries only pgcosmos extension, libraries, bootstrap SQL,
pg_hba, protected GUCs, plugin names, gateway contract and owner kind.
The only DocumentDB string allowed is the documented
documentDbCredentialSecret plugin key.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
Go internal/ packages cannot be imported from another module, so
pkg/core re-exports the intent/profile types and wraps the renderer,
sync, parameter merge, resource split/validation and extension
lifecycle helpers. It adds no behavior. Verified by building and
running a throwaway external module that uses a replace directive.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
Adds a design doc for the ProductProfile/ClusterIntent seam, the
pkg/core facade, retained wire contracts and follow-ups; documents the
new sidecar-injector gateway parameters; points AGENTS.md at the core;
adds an Unreleased CHANGELOG entry.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
Full validation for the product-neutral core: every Go module builds
and vets, unit tests pass (with -race for the operator and sidecar
injector), and golangci-lint reports no new findings against main.
Replaces the deprecated pointer helpers the new golden test used.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
…ng profile

Introduce the operational contract a product implements to run on shared reconcilers: ClusterSpec (read-only operational view), ClusterStatus (neutral status fields), NamingProfile (label/annotation/finalizer domain, service prefix, metric attribute) and the generic ClusterAdapter[T]. DocumentDBAdapter implements it with values identical to today's constants; util.MongoConnectionString is extracted from GenerateConnectionString.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
Extract util.BuildGatewayService (name, owner, port, environment) from GetDocumentDBServiceDefinition, and add otel.GenerateConfigMapDataWithOptions so the cluster resource attribute and generator header come from the product NamingProfile. DocumentDB output, including the OTel config hash, is unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
Move the DocumentDB cluster reconcile loop (service, RBAC, CNPG render/sync,
PV retention and recovery, OTel config, extension upgrade, status) into
internal/engine as ClusterReconciler[T] driven by product.ClusterAdapter.
Multi-region replication is wired through the optional ReplicationHooks.
DocumentDBReconciler keeps its fields and methods and delegates to the
engine; behavior, events and controller wiring are unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
Move gateway TLS provisioning (SelfSigned, Provided, CertManager) into
engine.CertificateReconciler[T]. DNS names use the product Service prefix
and status is written through the adapter. The DocumentDB
CertificateReconciler delegates; behavior is unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
Move PV labeling, schema-version stamping, reclaim policy and security
mount options into engine.PersistentVolumeReconciler[T]. The owner walk
PV -> PVC -> CNPG Cluster -> custom resource matches the new
NamingProfile.Kind. The DocumentDB PV controller delegates; behavior is
unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
Move backup and scheduled-backup reconciliation into
engine.BackupReconciler[T,B] and engine.ScheduledBackupReconciler[T,S,B].
Product Backup and ScheduledBackup CRs expose the shared preview spec and
status types through small adapters. Pure backup logic becomes dbpreview
package functions (NewCNPGBackup, UpdateBackupStatus, BackupExpirationTime,
NextScheduleTime, ScheduledBackupName) and the existing methods delegate.
The DocumentDB controllers delegate; behavior is unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
Move schema/binary version, image rollback, storage resize, restore
schema compatibility and the ImageVolume preflight into internal/validation,
parameterized by product Terms. DocumentDBValidator delegates; messages and
behavior are unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
Re-export ClusterReconciler, CertificateReconciler, PersistentVolumeReconciler,
Backup/ScheduledBackup reconcilers, adapters and shared admission validation
through pkg/core, and add a test driving a fake second product (Account CR)
through the cluster reconciler using only the public API.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
make manifests generate yields no diff; all modules vet; operator and
sidecar tests pass; lint findings unchanged versus origin/main.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
Expose replication state construction for external product hooks, reconcile gateway contract parameters and owned Service ports, and configure the PostgreSQL health metric through the product naming profile. Preserve DocumentDB defaults and add external-product regression coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
Exercise cluster, certificate, backup and scheduled-backup reconciliation, including client failures, replication hooks, retention and PV recovery. Cover public validation and resource helpers and SQL execution failures.

Changed-statement coverage is 94.55% with the CI coverage command and unchanged 90% threshold.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
hossain-rayhan and others added 3 commits October 22, 2026 11:34
Assert an immediate requeue through Result.IsZero and RequeueAfter while preserving the finalizer persistence check.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
Quote product-defined extension and replication role identifiers and catalog lookup literals with the existing PostgreSQL quoting helpers. Keep ImageVolume dry-run names independent of human-readable product names without changing the intentional fail-open policy.

Add regressions for hyphenated, mixed-case and quoted SQL names, and capability enforcement and caching with product display names containing spaces, punctuation and excessive length.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
Keep truncated Service names DNS-valid and derive certificate SANs from the actual Service name in both certificate modes. Cover both naming boundaries with focused regressions.

Reuse a realistic public Cosmos profile across cluster rendering and reconciliation, gateway contract convergence, credentials and TLS injection, admission, extension upgrades, and custom Backup/PV restore metadata. Verify the public fixture from an independent consumer module, and document destination-specific naming and plugin identity constraints.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>
@hossain-rayhan
hossain-rayhan force-pushed the developer/product-neutral-core branch from 7f656c3 to 4f4015a Compare October 22, 2026 18:37
@hossain-rayhan

Copy link
Copy Markdown
Collaborator Author

The product-neutral extraction is well structured, and I did not find an additional DocumentDB behavior regression while tracing the cluster, certificate, PV, backup, and scheduled-backup paths. The focused suites also pass (go test ./internal/product ./internal/cnpg ./internal/engine ./internal/validation ./pkg/core), consistent with the green golden and E2E checks.

I am requesting changes for the two open correctness issues in the new second-product surface:

  • Product-defined extension and role names are emitted as bare SQL identifiers, so valid hyphenated or mixed-case names break extension upgrades and replication grants: #482 (comment)
  • The ImageVolume probe derives generateName from the human-readable product name. A name such as Cosmos DB makes the baseline probe invalid and the intentionally fail-open path bypasses capability enforcement: #482 (comment)

Both should be fixed with focused regression tests before merging.

addressed with verification

func (n NamingProfile) ServiceName(clusterName string) string {
name := n.ServicePrefix + clusterName
if len(name) > 63 {
name = strings.TrimRight(name[:63], "-")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High - preserve uniqueness when shortening Service names. This truncates away the only differing suffix for valid CR names that share a long prefix. For example, with the DocumentDB prefix, aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-east and ...-west both become the same 63-character Service name. The second reconciliation finds the first cluster's Service; UpsertService intentionally does not mutate it because the owner UID differs, but returns it as success, so this cluster proceeds using the other cluster's Service IP and can publish a connection string routed to the wrong database. Please retain a deterministic hash suffix when shortening (and add a collision regression test) rather than plain truncation.

@xgerman xgerman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed current head 4f4015a. The two previous blockers are fixed correctly: SQL identifiers/literals are quoted and the ImageVolume probe now uses a fixed DNS-safe name. The expanded second-product lifecycle tests are useful, and the focused operator-core and injector suites pass. All CI checks are also green.

One new blocking correctness issue remains: plain truncation makes distinct valid cluster names collide on the same gateway Service. Because UpsertService preserves a differently owned Service but returns it successfully, the second cluster can use and publish the first cluster's endpoint. Please preserve uniqueness with a stable hash suffix and cover the collision case:

Plain truncation to 63 characters made cluster names that differ only past the cut share one gateway Service. UpsertService returned the other owner's Service as success, so the second cluster published the first cluster's endpoint.

Shorten long names with a stable FNV-32a hash of the cluster name instead. A cluster that already controls a plainly truncated Service from an earlier release keeps it, so existing endpoints and certificate DNS names do not change. UpsertService now refuses a same-named Service controlled by another owner, and certificate DNS names follow the resolved Service name.

Align the consumers that compute or look up the name: the e2e port-forward helper mirrors the hashed name, and kubectl documentdb status matches the Service by its controlling DocumentDB instead of a name substring that could select another cluster's Service.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 33b3cb3c-1b25-4ffb-bff1-ebf0f3d04058
Signed-off-by: Rayhan Hossain <hossain.rayhan@outlook.com>

@xgerman xgerman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed cd3071e. The previous Service-name collision is addressed: long names now get stable hash suffixes, foreign-owned Services are rejected, and an owned legacy Service keeps its endpoint. The original SQL quoting and ImageVolume probe fixes remain intact. Focused operator-core, kubectl CLI, and E2E port-forward helper suites pass; CI is green.

One medium-severity recovery regression remains: deleting a legacy long-name Service changes the resolved hostname to the hashed form, but the existing managed Certificate is never updated to cover it. I reproduced this with the fake client: create the legacy Service and its certificate, delete the Service, then rerun certificate reconciliation. The chosen new Service hostname is absent from the certificate's SANs. Please keep the chosen naming scheme stable across Service recreation, or reconcile certificate DNS names when the resolved Service name changes.

svc := &corev1.Service{}
if err := c.Get(ctx, types.NamespacedName{Name: legacy, Namespace: obj.GetNamespace()}, svc); err != nil {
if apierrors.IsNotFound(err) {
return name, nil

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium - keep TLS valid when a legacy Service is recreated. For an upgraded long-name cluster, the managed Certificate contains the legacy Service DNS names. If that Service is deleted, this branch switches the cluster to the hashed name, but both certificate paths only set Spec.DNSNames when initially creating the Certificate; an existing Certificate is left unchanged. Even explicitly rerunning certificate reconciliation does not repair it. I reproduced this with the fake client: the recreated hostname documentdb-service-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-2f5a7991.default.svc is absent from the existing legacy certificate's SANs. Strict TLS clients using Service DNS then fail hostname validation. Persist the chosen name so recreation retains it, or reconcile the existing certificate SANs when the resolved name changes, with a legacy-Service deletion/recreation regression test.

@xgerman

xgerman commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

Runtime-tested exact head cd3071e in local kind (Kubernetes 1.33.1, Linux arm64, containerd 2.1.0), with ImageVolume enabled on the API server and kubelet. Both operator and injector were built from the PR's Dockerfiles and deployed through the PR Helm chart.

Passed:

  • Two long-name DocumentDB clusters sharing the same first 44 characters reached healthy state with distinct hashed gateway Services, correct owner references/selectors, and different Service IPs.
  • Both endpoints passed strict TLS hostname/CA validation, SCRAM-SHA-256 authentication, and MongoDB-driver insert/read/update/delete checks. Using the same database/collection names with separate markers confirmed that requests reached separate databases.
  • The ImageVolume-mounted DocumentDB extension initialized (documentdb version 1.0-0); the shared reconciler granted documentdb_admin_role to streaming_replica.
  • Scaling one cluster from 1 to 2 instances created a physical standby (pg_is_in_recovery() = true), then scaling back to 1 succeeded.
  • Seeded an owned legacy Service/certificate to model a pre-upgrade long-name cluster. The PR preserved that legacy endpoint, and an actual TLS handshake against its legacy Service hostname succeeded.
  • Cluster deletion cleaned up its PVCs/PVs.

The reported TLS recovery defect is confirmed on the real cluster: deleting that legacy Service caused reconciliation to create the hashed Service, but the existing Certificate retained the legacy SANs (generation/revision unchanged at 2). An actual strict TLS connection using the new hashed Service hostname failed with x509: certificate is valid for [legacy names], not [hashed name]. Connecting to the same endpoint while verifying the legacy hostname succeeded, isolating this to stale certificate SANs rather than connectivity or trust. The resource still reported Cluster in healthy state.

Test resources were removed and the original released operator/injector restored. No source changes were made.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants