web: gracefully degrade Ontology Manager when /object-views route is absent#3
Open
wangmingzhou1986 wants to merge 1031 commits into
Open
web: gracefully degrade Ontology Manager when /object-views route is absent#3wangmingzhou1986 wants to merge 1031 commits into
wangmingzhou1986 wants to merge 1031 commits into
Conversation
ICA-2 — formalises the table lifecycle endpoints (list / create / load /
drop / rename) that already exist in the Go service so they have parity
tests and a shape mirroring the Rust source in
services/iceberg-catalog-service/src/{domain/table.rs,handlers/rest_catalog/tables.rs}.
Changes
- repo.Repo now wraps a DB interface so unit tests can drive it with
pgxmock instead of a live Postgres.
- CreateTable maps Postgres unique_violation (SQLSTATE 23505) to a
stable "already exists in namespace" error matching Rust's
TableError::AlreadyExists. The REST handler already maps this to 409.
- New repo_test.go covers the table CRUD lifecycle with pgxmock,
including a concurrent-create test that asserts only one of two
racing callers succeeds and the loser sees AlreadyExists.
- New tables_test.go covers the handler surface (auth, validation,
create→load→drop→404 round-trip, conflict→409, rename across namespaces).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Port `services/connector-management-service/src/connectors/bigquery.rs`
to the Go adapter contract introduced in CMA-0. The new
`internal/adapters/bigquery` package implements all four
[adapters.ConnectorAdapter] capabilities:
- DiscoverSources — list datasets + tables via the BigQuery v2 REST
API (`projects/{p}/datasets`, `…/datasets/{ds}/tables`), failing
soft on per-dataset listing errors so partial discovery still
surfaces.
- QueryVirtualTable — bounded SELECT … LIMIT preview that returns
JSON rows (mirrors Rust `bounded_preview`).
- StreamArrow — paginated `jobs.query` + `getQueryResults`
with `pageToken`, materialised as a single Arrow IPC frame backed
by apache/arrow-go/v18 (Utf8 columns, mirroring Rust's
`materialize_arrow_stream`).
- BuildIngestSpec — `dataset.table` selector → IngestSpec carrying
project_id / dataset_id / table_id (+ optional location / query)
under the `"bigquery"` source discriminator.
Auth flavours match Rust:
- `service_account_json` (JSON string OR object) → self-signed RS256
JWT exchanged at `oauth2.googleapis.com/token` for an access token.
- `access_token` → forwarded as a Bearer token.
`apiBase`, `tokenURL`, the embedded `*http.Client`, and the wallclock
used for JWT iat/exp are all overridable on the Adapter so tests can
stand up an httptest server with a deterministic clock.
Tests port the Rust `#[cfg(test)]` unit suite (config validation,
query builder, row/column extractors, service-account parsing) and
add a `fakeBigQuery` httptest server that exercises:
- `DiscoverSources` over a real REST round-trip with dataset + table
enumeration and bearer-token assertion;
- `QueryVirtualTable` over a single jobs.query response, asserting
the LIMIT-clamped preview SQL and decoded JSON rows;
- `StreamArrow` across three pages (jobs.query + two
`getQueryResults` followups) plus full Arrow IPC reader-side
decoding to confirm row count and schema parity;
- `obtainAccessToken` over a fake token endpoint with an RS256 JWT
that is parsed back with the matching public key and verified
against expected iss/aud/scope/iat/exp claims.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ships from Rust Mirrors services/tenancy-organizations-service/src/handlers/projects.rs: slug + workspace-slug normalisation are byte-exact; folder name and slug derivation match the Rust split_whitespace + ascii-alphanumeric walks; memberships upsert idempotently on (project_id, user_id); resource bindings keep (resource_kind, resource_id) as the conflict target so re-binding moves a resource between projects without orphan rows. The handler holds *pgxpool.Pool directly (Rust state.ontology_db analogue) and runs all SQL inline; routes mount under /api/v1/projects in server.go, with the access checks delegated to the existing internal/domain helpers (ListAccessibleProjects, EnsureProjectViewAccess, EnsureProjectEditAccess, EnsureResourceManageAccess). Migration 0005 adds ontology_projects + memberships + resources + folders so the service is self-contained against a single DATABASE_URL. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… Rust Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors the open-table flavour of services/connector-management-service/ src/connectors/s3.rs. Discovery surfaces inline iceberg_tables[] and delta_tables[] entries with the s3_<format>_table source kind, dedupes shared selectors, and returns a non-empty result. Validate rejects configs missing the url/bucket identity field or any inline open-table declaration. Query, Arrow streaming, and ingest spec return ErrNotImplemented; clients consume zero-copy via /iceberg/v1/* until the catalog_bridge HTTP runtime is ported. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…m Rust
Adds the OEA-4 spatial query engine (within / intersects / nearest /
buffer) and clustering (DBSCAN, K-Means) ports of
src/geospatial/geospatial_base/{handlers/features.rs, domain/engine/
spatial_query.rs, domain/engine/clustering.rs, models/spatial_index.rs}.
The route_features handler is left for OEA-5; only the predicate +
clustering surface is wired here. Routes are exposed via the existing
geospatial.AppState.Routes() but remain unmounted by the binary,
matching the Rust crate's #[allow(dead_code)] geospatial module.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…from Rust DV-12 brings the Rust data_asset_catalog catalog metadata surface into the Go service: lineage-aware effective markings, catalog enum constants, dataset audit emissions on Patch/Put and a unit suite covering the resolver contracts (direct, inheritance, cycles, dedupe, cache invalidation). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors services/connector-management-service/src/connectors/parquet.rs. Uses apache/arrow-go's pqarrow reader so virtual-table preview and Arrow IPC streaming are served directly from the adapter instead of deferring to lake materialisation. Validate, magic-byte checks, source label / file-name derivation, and the limit clamp ([1, 500]) follow the Rust contract. URLs are fetched via http.DefaultClient (overridable for tests); local paths use os.ReadFile. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
IRF-5 — Foundry "Reset stream" workflow. Mirrors the Rust handler at
services/ingestion-replication-service/src/event_streaming/handlers/
stream_views.rs. Rotates the stream's view RID atomically, retires the
previous active view (active=false, retired_at stamped), inserts a
fresh one with generation+1, and best-effort truncates the underlying
Kafka topic + resets consumer offsets via the streaming runtime.
POST /api/v1/streaming/streams/{id}:reset
* handlers/stream_views.go — claims/permission gate, INGEST-kind
guard, downstream-pipelines-active guard with force=true override,
metadata rotation via repo, hot-buffer rotation via runtime, audit
log emission.
* models/models.go — StreamView wire shape + StreamRIDFor / ViewRIDFor
helpers + ResetStreamRequest/Response + StreamKind constants.
* repo/repo.go — transactional ResetStream (load → retire → mint v7 →
insert new view → optional schema mirror) + DownstreamPipelinesActive
that no-ops gracefully until streaming_topologies lands.
* repo/migrations/20260507000003_streaming_stream_views.sql — table
+ partial unique index enforcing one active view per stream.
* runtime: ProductionStreamingRuntime.ResetStream re-creates the
Kafka topic (delete + provision) so all records are dropped and
consumer offsets become invalid.
Depends on IRF-4 (Kafka admin direct path).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors services/connector-management-service/src/connectors/csv.rs. Parses the CSV with stdlib encoding/csv, returns rows as JSON (string cells per Rust's behaviour), encodes the full file as a single Arrow IPC frame for StreamArrow, and emits a "csv" ingest descriptor. URL fetches use http.DefaultClient (overridable in tests); local paths use os.ReadFile. Discovery yields a synthetic single source whose display name is derived from the file basename (path or URL tail). Also widens the parquet adapter's display-name derivation to use the URL tail when no path is configured. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the runtime scan that hydrates `ontology.reindex.v1` records. Mirrors the `CassandraScanner` in services/reindex-coordinator-service/ src/scan.rs so the Go and Rust coordinators publish bit-for-bit identical batches during the cut-over: same two-query pattern (`objects_by_type` index lookup → `objects_by_id` hydrate, with the all-types path using `ALLOW FILTERING`), same opaque base64 page-state encoding, same soft-delete filter on hydration, same `scanned` counter that includes deleted ids. Tests: a `//go:build integration` suite drives a real `cassandra:5.0` testcontainer via libs/testing.BootCassandra to verify the scanner hits both tables, paginates via the opaque token, drops deleted rows on hydration, and skips orphan index entries; plain unit tests cover the paging-state encode/decode helpers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors `services/connector-management-service/src/connectors/tableau.rs` as a thin wrapper over a new shared `catalogbridge` package that ports the inline-or-remote tabular catalog plumbing from Rust's `connectors::catalog_bridge`. The bridge handles ValidateConfig, DiscoverSources, and QueryVirtualTable with parity for inline `tables/views/datasets/streams/reports/entities` catalogs and the `base_url` + `catalog_path` / `*_path_template` HTTP fan-out. Per CMA-11a, the connector binds `tableau_view` as the default source kind and `site_id` as the identity field; StreamArrow / BuildIngestSpec return ErrNotImplemented because the Rust connector does not expose those capabilities. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors services/connector-management-service/src/connectors/json.rs
and lifts the NDJSON limitation: parseRows uses a streaming
json.Decoder so single objects, arrays, and one-value-per-line streams
are all surfaced as rows. CountRows aligns with the same decoder so an
NDJSON file whose first byte is '{' counts every record (Rust returns 1
in that case because it short-circuits on the first character).
StreamArrow encodes the parsed rows as a single Utf8-typed Arrow IPC
frame; BuildIngestSpec emits a "json" descriptor carrying url/path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors `services/connector-management-service/src/connectors/power_bi.rs` as a thin wrapper over the shared `catalogbridge` package: binds `power_bi_dataset` as the default source kind and `workspace_id` as the identity field for resource-template configs. Inline `datasets` catalogs short-circuit the HTTP path; remote `dataset_path_template` configs flow through the bridge's HTTP client. StreamArrow and BuildIngestSpec return ErrNotImplemented because the Rust connector does not expose those capabilities. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors `services/connector-management-service/src/connectors/odbc.rs` as a thin wrapper over the shared `catalogbridge` package: binds `odbc_table` as the default source kind and `dsn` as the identity field for resource-template configs. The Rust connector does not open ODBC handles in-process — it routes every read through the catalog-bridge HTTP path on the assumption that the connector-agent (or an inline `tables` catalog) speaks ODBC on the user's behalf. The Go port preserves that behaviour. The package doc records the gap: in-process ODBC in Go would require `github.com/alexbrainman/odbc` (unmaintained, Windows + macOS unixODBC only), which is intentionally not wired in here because CMA-11c is a wrapper-parity task. StreamArrow / BuildIngestSpec return ErrNotImplemented for the same reason as the other catalog-bridge connectors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors `services/connector-management-service/src/connectors/jdbc.rs` as a thin wrapper over the shared `catalogbridge` package: binds `jdbc_table` as the default source kind and `jdbc_url` as the identity field for resource-template configs. JDBC is a Java-only contract; the Rust side does not run JDBC drivers in-process either — every read flows through the catalog-bridge HTTP path on the assumption that the connector-agent runs the JDBC connection on the user's JVM, or that the inline `tables` catalog already carries the data. The Go port preserves that "JVM-out-of-process" model and records the gap in the package doc: in-process JDBC in Go either needs a JVM sidecar (the chosen approach in the broader migration plan, mirroring the existing python-sidecar pattern) or per-prefix translation to a native Go driver (`pgx`, `go-sql-driver/mysql`, `gosnowflake`, …). Both swaps are explicitly out of scope for CMA-11d. StreamArrow / BuildIngestSpec return ErrNotImplemented for the same reason as the other catalog-bridge connectors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… Rust Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the MySQL connector adapter as a Foundry-aligned thin wrapper over the shared tabular catalog bridge, mirroring `services/connector-management-service/src/connectors/mysql.rs`. Identity field is `host`, default source kind `mysql_table`. Both Rust `#[cfg(test)]` cases are ported and the standard discover/query/stream/ingest assertion set is added. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… Rust Closes the RC-2 gaps so the Go ReindexRecord wire shape is byte-exact to the Rust producer: - `Deleted` json tag drops `omitempty`. Rust uses `#[serde(default)]` with no `skip_serializing_if`, so the field is always emitted; the legacy ontology-indexer `ObjectChangedV1` consumer relies on the field being present. - `extractEmbedding` now skips non-numeric entries instead of failing the whole record, matching the Rust loop over `Value::as_f64`. Uses `*float64` to distinguish JSON `null` (skip) from a real `0.0`. - Ports the Rust `#[cfg(test)]` cases that were missing on the Go side: empty-array embedding collapses to nil, integer embeddings are accepted, heterogeneous arrays keep only the numeric entries, and a round-trip JSON shape test pinning the wire keys against `services/ontology-indexer::ObjectChangedV1`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…r from Rust Ports the Foundry "Branch retention" doc resolver and hourly archive worker from Rust src/domain/retention.rs and src/domain/retention_worker.rs: - internal/domain/retention: pure ParsePolicy/PolicyAsString, ResolveEffective (walks INHERITED up the parent chain, falls back to FOREVER on missing/cyclic ancestors), and IsArchiveEligible (skips roots, branches with OPEN transactions, and already-archived rows). - internal/runtime/retention: Worker with injectable Clock, Store, gauge and counter shims; RunOnce resolves + archives + sets the eligibility gauge; RunLoop skips the immediate first tick and defaults to one-hour cadence. RepoStore adapter delegates to ListRetentionCandidates and ArchiveBranchForRetentionWithOutbox. - cmd/main.go starts the loop in-process when RETENTION_WORKER_ENABLED is set; RETENTION_WORKER_INTERVAL overrides the cadence. - All seven #[cfg(test)] cases from domain/retention.rs ported verbatim plus cycle/non-positive-TTL/already-archived edge cases. Worker tests drive a fake store + fake clock for archive correctness, idempotency, error propagation, INHERITED chain resolution, and RunLoop start/stop. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…l/ldap/mssql/oracle/sftp) from Rust Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…from Rust Closes the third utility package called out in CMA-13 alongside the already-landed catalogbridge (CMA-11a) and opentable (CMA-9) ports: - internal/adapters/httpruntime/runtime.go — Client.Get/PostJSON/PostForm, agent-proxy hop, JSONBody/HeaderMap, mirroring src/connectors/http_runtime.rs - internal/adapters/httpruntime/egress.go — EgressPolicy + EgressPolicyFromState + ValidateURL, mirroring src/domain/egress.rs - ports the egress.rs #[cfg(test)] tests plus roundtrip tests for Get/PostJSON/PostForm/agent-proxy and HeaderMap/JSONBody helpers - updates connector-management-service parity row for runtime bridges Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors `services/connector-management-service/src/connectors/snowflake.rs`
(531 LOC) as a productive REST adapter against Snowflake's
`/api/v2/statements` endpoint. Both Rust auth flavours are preserved:
- Keypair JWT (preferred): RS256 over PKCS#8 RSA key, with `iss`/`sub`
encoding the `{account}.{USER}` identity exactly like Rust
(`obtain_token` mirrored 1:1 — only the user is uppercased; the
account is just trim+trailing-dot-normalised).
- OAuth bearer: passed through as `Authorization: Bearer …` with
`X-Snowflake-Authorization-Token-Type: OAUTH`.
Capabilities:
- DiscoverSources runs `SHOW TABLES IN SCHEMA :db.:schema` and emits
one DiscoveredSource per row, selectors qualified `db.schema.name`.
- QueryVirtualTable: `SELECT * FROM <qualified> LIMIT n` with limit
clamped into [1, 500] just like Rust.
- StreamArrow: full multi-partition fetch over `partitionInfo[]` capped
at MAX_PARTITIONS=50, materialised into a single Arrow IPC frame via
the same nullable-Utf8 schema as `materialize_arrow_stream` in
`connectors/mod.rs`.
- BuildIngestSpec: structural placeholder (account/database/schema/
table + optional warehouse/role/query) discriminated as "snowflake",
matching the CMA-0 contract until the bridge gains a typed variant.
The slice description suggests the Go port should use the gosnowflake
driver, but the Rust source materialises rows from REST JSON envelopes;
keeping the REST path here preserves the wire shape (HTTP envelopes,
source signatures, partition fan-out cap) byte-identical for parity.
Tests port the four Rust `#[cfg(test)]` cases (validate_config,
build_query qualification + passthrough, extract_rows column alignment,
base_url account subdomain) and add a httptest-driven `/api/v2/statements`
fake that exercises:
- DiscoverSources end-to-end with bearer/header-kind assertion;
- QueryVirtualTable shape including `parameters.ROWS_PER_RESULTSET`;
- StreamArrow multi-partition fan-out with Arrow IPC reader decoding
back to confirm 3 rows × 2 cols arrived through 1 POST + 2 GETs;
- HTTP-error surfacing as `Snowflake statements returned HTTP 403`.
- JWT signing path with a fixed clock so iat/exp/iss/sub claims are
deterministically asserted via the matching public key.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors `services/connector-management-service/src/connectors/kafka.rs` as a thin adapter over the shared `libs/event-bus-control` (controlbus) helpers — same package the ingestion-replication and edge-gateway services already use. Catalog-backed mode (parse `topics[]` from the connection config, return preview/sample messages) is always available. When `bootstrap_servers` (or the `brokers` alias) is configured, discovery upgrades to the live broker probe (segmentio/kafka-go via controlbus.DiscoverTopics) and virtual-table preview tails the last N messages off partition 0 (controlbus.TailMessages). This mirrors Rust's `cfg(feature = \"kafka-rdkafka\")` gated path: librdkafka in Rust, pure-Go kafka-go on the Go side. Public surface (1:1 with Rust): - ValidateConfig ← validate_config - Adapter.TestConnection ← test_connection - Adapter.DiscoverSources ← discover_sources - Adapter.QueryVirtualTable ← query_virtual_table - Adapter.FetchDataset ← fetch_dataset StreamArrow + BuildIngestSpec return adapters.ErrNotImplemented because Rust's kafka.rs ships neither — Arrow streaming and ingest-spec construction live in dedicated modules outside the read-side connector contract. Tests cover the three Rust `#[cfg(test)]` cases verbatim (parses_string_and_object_topics, validates_required_bootstrap_servers, finds_configured_topic) plus extras: limit clamping (1, 500, default 50), JSON sync payload (file_name sanitisation, source_signature splice), and the ErrNotImplemented surface for the two unsupported capabilities. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…es-to-latest-stable chore(web): update PostCSS to 8.5.14
…against-palantir-documentation Add migration parity checklists: AIP Agents/Threads/Assist and Compute Modules
…latest-stable-versions chore: bump Go module version to 1.25.1
…nd-by-govulncheck build: update Go security dependencies
…-and-content Revamp README with branding, badges, quickstart, and updated developer commands
…e-relationship-task-uhzur3 Enhance ontology manager: Object Views, property/value-type metadata, link & binding improvements, groups, and save-changes workflow
…ng-data-connection-checklist Data Connection: registry, streaming, webhooks, egress, credentials, UI enhancements, and tests
…c-files Add AIP Logic authoring UI, validation, preview runtime, and server endpoints
…bsent Ontology Manager loads object views inside a Promise.all with other resources. When the gateway has no route for /api/v1/object-views (and no backend implements it), the 404 unknown_service_route rejection propagates out of Promise.all and the whole page renders an error instead of content. Object views are optional: the page already builds client-side coreViews. Treat a 404 as an empty list so the page renders; when a backend implements the endpoint the catch never fires.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Opening Ontology Manager renders a red
unknown service routeerror instead of the page.OntologyManagerPage.refresh()loads resources withPromise.all([...]), one of which islistObjectViews()→GET /api/v1/object-views. The edge-gateway router table has no case for/api/v1/object-views(it is not under the/api/v1/ontologyprefix that owns the catch-all), andno backend implements it, so the request returns
404 {"code":"unknown_service_route"}. That singlerejection propagates out of
Promise.all, so the entire Ontology Manager fails to render — eventhough object views are an optional enhancement and the page already builds client-side
coreViewsfrom object/link types.
Fix
Treat a
404from/object-viewsas an empty list inlistObjectViews(), so the page renders withcoreViews. When a backend later implements the endpoint (returns 200), thecatchnever fires — nobehavior change. One file, frontend-only, no environment-specific content.
Verify
/ontology-managershowsunknown service route, page blank.Sandbox Ontology metadata + client-derived core Object Views render.