Fix web UI + AIP copilot wiring (4 independent bugs)#1
Open
wangmingzhou1986 wants to merge 1034 commits into
Open
Fix web UI + AIP copilot wiring (4 independent bugs)#1wangmingzhou1986 wants to merge 1034 commits into
wangmingzhou1986 wants to merge 1034 commits into
Conversation
…ships from Rust Mirrors services/tenancy-organizations-service/src/handlers/projects.rs: slug + workspace-slug normalisation are byte-exact; folder name and slug derivation match the Rust split_whitespace + ascii-alphanumeric walks; memberships upsert idempotently on (project_id, user_id); resource bindings keep (resource_kind, resource_id) as the conflict target so re-binding moves a resource between projects without orphan rows. The handler holds *pgxpool.Pool directly (Rust state.ontology_db analogue) and runs all SQL inline; routes mount under /api/v1/projects in server.go, with the access checks delegated to the existing internal/domain helpers (ListAccessibleProjects, EnsureProjectViewAccess, EnsureProjectEditAccess, EnsureResourceManageAccess). Migration 0005 adds ontology_projects + memberships + resources + folders so the service is self-contained against a single DATABASE_URL. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… Rust Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors the open-table flavour of services/connector-management-service/ src/connectors/s3.rs. Discovery surfaces inline iceberg_tables[] and delta_tables[] entries with the s3_<format>_table source kind, dedupes shared selectors, and returns a non-empty result. Validate rejects configs missing the url/bucket identity field or any inline open-table declaration. Query, Arrow streaming, and ingest spec return ErrNotImplemented; clients consume zero-copy via /iceberg/v1/* until the catalog_bridge HTTP runtime is ported. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…m Rust
Adds the OEA-4 spatial query engine (within / intersects / nearest /
buffer) and clustering (DBSCAN, K-Means) ports of
src/geospatial/geospatial_base/{handlers/features.rs, domain/engine/
spatial_query.rs, domain/engine/clustering.rs, models/spatial_index.rs}.
The route_features handler is left for OEA-5; only the predicate +
clustering surface is wired here. Routes are exposed via the existing
geospatial.AppState.Routes() but remain unmounted by the binary,
matching the Rust crate's #[allow(dead_code)] geospatial module.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…from Rust DV-12 brings the Rust data_asset_catalog catalog metadata surface into the Go service: lineage-aware effective markings, catalog enum constants, dataset audit emissions on Patch/Put and a unit suite covering the resolver contracts (direct, inheritance, cycles, dedupe, cache invalidation). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors services/connector-management-service/src/connectors/parquet.rs. Uses apache/arrow-go's pqarrow reader so virtual-table preview and Arrow IPC streaming are served directly from the adapter instead of deferring to lake materialisation. Validate, magic-byte checks, source label / file-name derivation, and the limit clamp ([1, 500]) follow the Rust contract. URLs are fetched via http.DefaultClient (overridable for tests); local paths use os.ReadFile. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
IRF-5 — Foundry "Reset stream" workflow. Mirrors the Rust handler at
services/ingestion-replication-service/src/event_streaming/handlers/
stream_views.rs. Rotates the stream's view RID atomically, retires the
previous active view (active=false, retired_at stamped), inserts a
fresh one with generation+1, and best-effort truncates the underlying
Kafka topic + resets consumer offsets via the streaming runtime.
POST /api/v1/streaming/streams/{id}:reset
* handlers/stream_views.go — claims/permission gate, INGEST-kind
guard, downstream-pipelines-active guard with force=true override,
metadata rotation via repo, hot-buffer rotation via runtime, audit
log emission.
* models/models.go — StreamView wire shape + StreamRIDFor / ViewRIDFor
helpers + ResetStreamRequest/Response + StreamKind constants.
* repo/repo.go — transactional ResetStream (load → retire → mint v7 →
insert new view → optional schema mirror) + DownstreamPipelinesActive
that no-ops gracefully until streaming_topologies lands.
* repo/migrations/20260507000003_streaming_stream_views.sql — table
+ partial unique index enforcing one active view per stream.
* runtime: ProductionStreamingRuntime.ResetStream re-creates the
Kafka topic (delete + provision) so all records are dropped and
consumer offsets become invalid.
Depends on IRF-4 (Kafka admin direct path).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors services/connector-management-service/src/connectors/csv.rs. Parses the CSV with stdlib encoding/csv, returns rows as JSON (string cells per Rust's behaviour), encodes the full file as a single Arrow IPC frame for StreamArrow, and emits a "csv" ingest descriptor. URL fetches use http.DefaultClient (overridable in tests); local paths use os.ReadFile. Discovery yields a synthetic single source whose display name is derived from the file basename (path or URL tail). Also widens the parquet adapter's display-name derivation to use the URL tail when no path is configured. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the runtime scan that hydrates `ontology.reindex.v1` records. Mirrors the `CassandraScanner` in services/reindex-coordinator-service/ src/scan.rs so the Go and Rust coordinators publish bit-for-bit identical batches during the cut-over: same two-query pattern (`objects_by_type` index lookup → `objects_by_id` hydrate, with the all-types path using `ALLOW FILTERING`), same opaque base64 page-state encoding, same soft-delete filter on hydration, same `scanned` counter that includes deleted ids. Tests: a `//go:build integration` suite drives a real `cassandra:5.0` testcontainer via libs/testing.BootCassandra to verify the scanner hits both tables, paginates via the opaque token, drops deleted rows on hydration, and skips orphan index entries; plain unit tests cover the paging-state encode/decode helpers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors `services/connector-management-service/src/connectors/tableau.rs` as a thin wrapper over a new shared `catalogbridge` package that ports the inline-or-remote tabular catalog plumbing from Rust's `connectors::catalog_bridge`. The bridge handles ValidateConfig, DiscoverSources, and QueryVirtualTable with parity for inline `tables/views/datasets/streams/reports/entities` catalogs and the `base_url` + `catalog_path` / `*_path_template` HTTP fan-out. Per CMA-11a, the connector binds `tableau_view` as the default source kind and `site_id` as the identity field; StreamArrow / BuildIngestSpec return ErrNotImplemented because the Rust connector does not expose those capabilities. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors services/connector-management-service/src/connectors/json.rs
and lifts the NDJSON limitation: parseRows uses a streaming
json.Decoder so single objects, arrays, and one-value-per-line streams
are all surfaced as rows. CountRows aligns with the same decoder so an
NDJSON file whose first byte is '{' counts every record (Rust returns 1
in that case because it short-circuits on the first character).
StreamArrow encodes the parsed rows as a single Utf8-typed Arrow IPC
frame; BuildIngestSpec emits a "json" descriptor carrying url/path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors `services/connector-management-service/src/connectors/power_bi.rs` as a thin wrapper over the shared `catalogbridge` package: binds `power_bi_dataset` as the default source kind and `workspace_id` as the identity field for resource-template configs. Inline `datasets` catalogs short-circuit the HTTP path; remote `dataset_path_template` configs flow through the bridge's HTTP client. StreamArrow and BuildIngestSpec return ErrNotImplemented because the Rust connector does not expose those capabilities. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors `services/connector-management-service/src/connectors/odbc.rs` as a thin wrapper over the shared `catalogbridge` package: binds `odbc_table` as the default source kind and `dsn` as the identity field for resource-template configs. The Rust connector does not open ODBC handles in-process — it routes every read through the catalog-bridge HTTP path on the assumption that the connector-agent (or an inline `tables` catalog) speaks ODBC on the user's behalf. The Go port preserves that behaviour. The package doc records the gap: in-process ODBC in Go would require `github.com/alexbrainman/odbc` (unmaintained, Windows + macOS unixODBC only), which is intentionally not wired in here because CMA-11c is a wrapper-parity task. StreamArrow / BuildIngestSpec return ErrNotImplemented for the same reason as the other catalog-bridge connectors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors `services/connector-management-service/src/connectors/jdbc.rs` as a thin wrapper over the shared `catalogbridge` package: binds `jdbc_table` as the default source kind and `jdbc_url` as the identity field for resource-template configs. JDBC is a Java-only contract; the Rust side does not run JDBC drivers in-process either — every read flows through the catalog-bridge HTTP path on the assumption that the connector-agent runs the JDBC connection on the user's JVM, or that the inline `tables` catalog already carries the data. The Go port preserves that "JVM-out-of-process" model and records the gap in the package doc: in-process JDBC in Go either needs a JVM sidecar (the chosen approach in the broader migration plan, mirroring the existing python-sidecar pattern) or per-prefix translation to a native Go driver (`pgx`, `go-sql-driver/mysql`, `gosnowflake`, …). Both swaps are explicitly out of scope for CMA-11d. StreamArrow / BuildIngestSpec return ErrNotImplemented for the same reason as the other catalog-bridge connectors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… Rust Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the MySQL connector adapter as a Foundry-aligned thin wrapper over the shared tabular catalog bridge, mirroring `services/connector-management-service/src/connectors/mysql.rs`. Identity field is `host`, default source kind `mysql_table`. Both Rust `#[cfg(test)]` cases are ported and the standard discover/query/stream/ingest assertion set is added. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… Rust Closes the RC-2 gaps so the Go ReindexRecord wire shape is byte-exact to the Rust producer: - `Deleted` json tag drops `omitempty`. Rust uses `#[serde(default)]` with no `skip_serializing_if`, so the field is always emitted; the legacy ontology-indexer `ObjectChangedV1` consumer relies on the field being present. - `extractEmbedding` now skips non-numeric entries instead of failing the whole record, matching the Rust loop over `Value::as_f64`. Uses `*float64` to distinguish JSON `null` (skip) from a real `0.0`. - Ports the Rust `#[cfg(test)]` cases that were missing on the Go side: empty-array embedding collapses to nil, integer embeddings are accepted, heterogeneous arrays keep only the numeric entries, and a round-trip JSON shape test pinning the wire keys against `services/ontology-indexer::ObjectChangedV1`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…r from Rust Ports the Foundry "Branch retention" doc resolver and hourly archive worker from Rust src/domain/retention.rs and src/domain/retention_worker.rs: - internal/domain/retention: pure ParsePolicy/PolicyAsString, ResolveEffective (walks INHERITED up the parent chain, falls back to FOREVER on missing/cyclic ancestors), and IsArchiveEligible (skips roots, branches with OPEN transactions, and already-archived rows). - internal/runtime/retention: Worker with injectable Clock, Store, gauge and counter shims; RunOnce resolves + archives + sets the eligibility gauge; RunLoop skips the immediate first tick and defaults to one-hour cadence. RepoStore adapter delegates to ListRetentionCandidates and ArchiveBranchForRetentionWithOutbox. - cmd/main.go starts the loop in-process when RETENTION_WORKER_ENABLED is set; RETENTION_WORKER_INTERVAL overrides the cadence. - All seven #[cfg(test)] cases from domain/retention.rs ported verbatim plus cycle/non-positive-TTL/already-archived edge cases. Worker tests drive a fake store + fake clock for archive correctness, idempotency, error propagation, INHERITED chain resolution, and RunLoop start/stop. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…l/ldap/mssql/oracle/sftp) from Rust Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…from Rust Closes the third utility package called out in CMA-13 alongside the already-landed catalogbridge (CMA-11a) and opentable (CMA-9) ports: - internal/adapters/httpruntime/runtime.go — Client.Get/PostJSON/PostForm, agent-proxy hop, JSONBody/HeaderMap, mirroring src/connectors/http_runtime.rs - internal/adapters/httpruntime/egress.go — EgressPolicy + EgressPolicyFromState + ValidateURL, mirroring src/domain/egress.rs - ports the egress.rs #[cfg(test)] tests plus roundtrip tests for Get/PostJSON/PostForm/agent-proxy and HeaderMap/JSONBody helpers - updates connector-management-service parity row for runtime bridges Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors `services/connector-management-service/src/connectors/snowflake.rs`
(531 LOC) as a productive REST adapter against Snowflake's
`/api/v2/statements` endpoint. Both Rust auth flavours are preserved:
- Keypair JWT (preferred): RS256 over PKCS#8 RSA key, with `iss`/`sub`
encoding the `{account}.{USER}` identity exactly like Rust
(`obtain_token` mirrored 1:1 — only the user is uppercased; the
account is just trim+trailing-dot-normalised).
- OAuth bearer: passed through as `Authorization: Bearer …` with
`X-Snowflake-Authorization-Token-Type: OAUTH`.
Capabilities:
- DiscoverSources runs `SHOW TABLES IN SCHEMA :db.:schema` and emits
one DiscoveredSource per row, selectors qualified `db.schema.name`.
- QueryVirtualTable: `SELECT * FROM <qualified> LIMIT n` with limit
clamped into [1, 500] just like Rust.
- StreamArrow: full multi-partition fetch over `partitionInfo[]` capped
at MAX_PARTITIONS=50, materialised into a single Arrow IPC frame via
the same nullable-Utf8 schema as `materialize_arrow_stream` in
`connectors/mod.rs`.
- BuildIngestSpec: structural placeholder (account/database/schema/
table + optional warehouse/role/query) discriminated as "snowflake",
matching the CMA-0 contract until the bridge gains a typed variant.
The slice description suggests the Go port should use the gosnowflake
driver, but the Rust source materialises rows from REST JSON envelopes;
keeping the REST path here preserves the wire shape (HTTP envelopes,
source signatures, partition fan-out cap) byte-identical for parity.
Tests port the four Rust `#[cfg(test)]` cases (validate_config,
build_query qualification + passthrough, extract_rows column alignment,
base_url account subdomain) and add a httptest-driven `/api/v2/statements`
fake that exercises:
- DiscoverSources end-to-end with bearer/header-kind assertion;
- QueryVirtualTable shape including `parameters.ROWS_PER_RESULTSET`;
- StreamArrow multi-partition fan-out with Arrow IPC reader decoding
back to confirm 3 rows × 2 cols arrived through 1 POST + 2 GETs;
- HTTP-error surfacing as `Snowflake statements returned HTTP 403`.
- JWT signing path with a fixed clock so iat/exp/iss/sub claims are
deterministically asserted via the matching public key.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors `services/connector-management-service/src/connectors/kafka.rs` as a thin adapter over the shared `libs/event-bus-control` (controlbus) helpers — same package the ingestion-replication and edge-gateway services already use. Catalog-backed mode (parse `topics[]` from the connection config, return preview/sample messages) is always available. When `bootstrap_servers` (or the `brokers` alias) is configured, discovery upgrades to the live broker probe (segmentio/kafka-go via controlbus.DiscoverTopics) and virtual-table preview tails the last N messages off partition 0 (controlbus.TailMessages). This mirrors Rust's `cfg(feature = \"kafka-rdkafka\")` gated path: librdkafka in Rust, pure-Go kafka-go on the Go side. Public surface (1:1 with Rust): - ValidateConfig ← validate_config - Adapter.TestConnection ← test_connection - Adapter.DiscoverSources ← discover_sources - Adapter.QueryVirtualTable ← query_virtual_table - Adapter.FetchDataset ← fetch_dataset StreamArrow + BuildIngestSpec return adapters.ErrNotImplemented because Rust's kafka.rs ships neither — Arrow streaming and ingest-spec construction live in dedicated modules outside the read-side connector contract. Tests cover the three Rust `#[cfg(test)]` cases verbatim (parses_string_and_object_topics, validates_required_bootstrap_servers, finds_configured_topic) plus extras: limit clamping (1, 500, default 50), JSON sync payload (file_name sanitisation, source_signature splice), and the ErrNotImplemented surface for the two unsupported capabilities. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes RC-3 partial: builds JobRepo on top of the existing
0001_reindex_jobs migration. Mirrors src/state.rs::repo and
src/main.rs::PROCESSED_EVENTS_TABLE 1:1.
JobRepo methods (UpsertQueued, Load, ListResumable, MarkRunning,
Advance, MarkTerminal) gate every write on the previous status in
the WHERE clause so concurrent coordinator replicas produce at most
one successful UPDATE; the loser surfaces *state.IllegalTransitionError.
Self-loops on terminal rows are no-ops to keep Kafka redelivery safe.
NewProcessedEventsStore wires libs/idempotency.PgStore to the
reindex_coordinator.processed_events table — the Kafka consumer
slice (RC-5) records the deterministic batch event_id BEFORE
producing each batch, so a crash between "produce batch" and
"advance resume_token" replays without double-publishing.
Tests:
- repo_test.go: unit tests for JobRecord helpers + dedup wiring
constants (no DB, run in default build).
- repo_integration_test.go (//go:build integration): testcontainers
Postgres exercising upsert idempotency, the full state machine,
Advance counter accumulation, ListResumable ordering, and
processed_events dedup across PgStore instances.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors handlers::rest_catalog::config::get_config and handlers::diagnose::run_diagnose
to the Go iceberg-catalog-service:
- GET /iceberg/v1/config returns {defaults: {warehouse}, overrides: {}} with the
warehouse URI plumbed from the existing config.WarehouseURI.
- POST /iceberg/v1/diagnose runs the same two-step probe as Rust (list_namespaces
then load_probe_namespace against `_diagnostic`) with byte-exact JSON shape:
{client, success, steps[{name, ok, latency_ms, detail}], total_latency_ms}.
- Step 2 stays soft-warn (ok=true) when the probe namespace is missing, exactly
like Rust.
To back the probe steps the Store interface gains ListTopLevelNamespaces (mirror
of namespace::list with parent=None) and FetchNamespaceByName (mirror of
namespace::fetch). Repo implements both against iceberg_namespaces; Handlers
gains a WarehouseURI field wired from cfg.WarehouseURI in main.go. The Rust
metrics::record_rest_request and audit::diagnose_executed hooks are deferred
to ICA-10 (audit + observability slice).
Tests cover auth gating, JSON shape, success path, default project_rid, list
errors, missing probe namespace soft-warn, and the existing fakeAppendStore is
extended with the two new Store methods so the existing append handler suite
stays green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…against-palantir-documentation Add migration parity checklists: AIP Agents/Threads/Assist and Compute Modules
…latest-stable-versions chore: bump Go module version to 1.25.1
…nd-by-govulncheck build: update Go security dependencies
…-and-content Revamp README with branding, badges, quickstart, and updated developer commands
…e-relationship-task-uhzur3 Enhance ontology manager: Object Views, property/value-type metadata, link & binding improvements, groups, and save-changes workflow
…ng-data-connection-checklist Data Connection: registry, streaming, webhooks, egress, credentials, UI enhancements, and tests
…c-files Add AIP Logic authoring UI, validation, preview runtime, and server endpoints
…taset_views An earlier migration may have created dataset_views with a legacy (materialized-view) shape that lacks the version-control columns (branch_id, head_transaction_id, computed_at, ...). CREATE TABLE IF NOT EXISTS then no-ops, so the later CREATE INDEX on dataset_views(branch_id) fails with "column branch_id does not exist" on such databases. Add IF NOT EXISTS column reconciliation and relax legacy NOT NULL columns so the migration re-runs idempotently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix web UI + AIP copilot wiring (4 independent bugs)
Found while bringing up OpenFoundry on a single-node k3s cluster and validating a
gene-protein ontology end-to-end. Four small, isolated upstream bugs blocked the web
UI and the AIP copilot. Branch: fix/web-copilot-ontology-wiring (4 commits).
1. dataset-versioning: versioning_init not idempotent vs legacy dataset_views
20260501000001_versioning_init.sql.
(materialized-view) shape lacking VC columns; CREATE TABLE IF NOT EXISTS no-ops,
then CREATE INDEX on dataset_views(branch_id) fails.
already used for dataset_transactions in this file).
2. agent-runtime: copilot/chat never bind a real LLM provider
enabled rows in ai_providers.
fake and never reads ai_providers.
3. ontology-definition: ListObjectTypes returns wrong list envelope
interfaces/links/groups handlers use {"data":[...],total,page,per_page}.
4. agent-runtime: copilot/chat unreachable under gateway /api/v1/ai prefix
routes /api/v1/ai here without rewrite, but the service only mounts
/api/v1/agent-runtime.
Verification
copilot/ask returns a real answer ("TP53 encodes p53, a tumor suppressor ...").