Skip to content

Security: per-ontology access checks, upload limits, DB connection sources, fixes for #37 #38 #39 - #41

Merged
jingw2 merged 27 commits into
masterfrom
dev
Nov 13, 2026
Merged

jingw2 merged 27 commits into
masterfrom
dev

Conversation

@jingw2

@jingw2 jingw2 commented Nov 13, 2026

Copy link
Copy Markdown
Owner

⚠️ Security fix — please merge first

On current master, the legacy v1 ontology routes (entities, logic, actions, graph, files, export, audit, extraction, GET/PUT/DELETE /ontologies/{id}) and the classic v2 graph, search and logic/action routes only check that the caller is logged in. Any account, including a self-registered viewer, can:

  1. get ontology ids from /api/v1/overview/stats, then
  2. read or export any ontology with /api/v1/ontologies/{id}/export; editors can also modify ontologies they have no grant on.

3b5be6e makes every one of these routes check the caller's ontology_project_access_grants row and answer 404 ONTOLOGY_NOT_FOUND otherwise. Overview stats only count ontologies the caller can discover, and Neo4j traversals stay inside the requested ontology. tests/agent/test_ontology_route_access.py reproduces the leak: 14 of its cases fail on master and pass here.

🚀 Deploy step

After deploying, run once:

DATABASE_URL=... python backend/scripts/backfill_creator_grants.py

Ontologies inserted by direct SQL have no creator grant, so until this runs nobody can open them, including admins. Migration 0003 already backfilled ontologies that existed when it ran.

Issue fixes

Other changes in this PR

From this round:

  • 8faa937 uploads are read in chunks with a hard size limit, and partial files are cleaned up on failure.
  • 8c27560 pipeline and refresh sources use saved encrypted DB connections; inline SQL credentials are dropped on save. Adds GET /connections/{id}/resources and /resources/{resource}/sample.
  • 4d6080c model deletion works on schemas built without the versioning migration.
  • 7d7017e supported Python is >=3.11,<3.13.
  • 8bf5e51 Chroma uses port 8000 inside the compose network; README says React 19.
  • 3b39ad3 the snapshot-freshness test reads the clock per test.
  • cc201c4 removes synchronous setState inside effects (react-hooks lint).

Earlier dev commits not yet on master: extraction improvements (df96044…44f8fe7, 405c485), the delete cleanup in dd10d6d, 8addf59, 28f2a9c and c2831aa.

Testing

  • Backend full suite: 2083 passed, 62 skipped, 1 failed. The failure is test_acceptance_matrix; it needs runtime fixture databases and the ontexus_runtime package, which aren't set up locally, and is unrelated to these changes. test_transport_parity.py was excluded because it needs that same package.
  • Each commit in this round was checked out in a clean worktree and its own tests run there.
  • Frontend: 229 tests passed, tsc is clean, and eslint reports no errors.

Closes #37
Closes #38
Closes #39

🤖 Generated with Claude Code

jingw2 and others added 27 commits September 14, 2026 22:36
…/enum values

Two concrete failure modes in the 简易LLM ontology build path: business
rules (e.g. "促销定价规则") got extracted as entities instead of
logic_rules, and enumeration values of a concept's property (信用分层's
"A层"/"B层", 逾期阶段's "M0"/"M2+", 融资轮次's "D轮融资") got extracted
as standalone entities/instances instead of being recognized as values,
not concepts. The ontology's entity layer is meant to hold unified
concept-level types only (per graph-engineering discipline already
partially applied here via CONCEPT_INSTANCE_DIRECTIVE), and neither
existing directive's examples (named objects: company/person/order
names) covered these two shapes.

Two-layer fix, matching the existing prompt+deterministic-backstop
pattern:
- Prompt layer: explicit negative rules added to both
  CONCEPT_INSTANCE_DIRECTIVE and the shared extract_ontology user
  message, with concrete examples of both failure modes.
- Backstop layer: normalize_extracted_ontology now drops entities
  matching _looks_like_rule_name (business-rule/policy suffixes) or
  _looks_like_enum_value (tier/stage/round-marker shape) — the same
  place junk entity names were already being dropped. Relations left
  dangling by a dropped entity are removed by the existing
  dangling-reference filter, no extra handling needed.

Also extends the document-extraction eval harness
(evals/document_extraction/validators.py) with the same enum-value
pattern in instance_leakage_gate and a new rule_leakage_gate, so this
regression class is caught automatically going forward — the existing
_ID_PATTERN/_TRAILING_DIGITS_PATTERN gate only caught bare-digit IDs
(CUS0001, 客户001), not letter+marker shapes like "A层" or "M2+".

Updates the pre-existing test that had encoded the old (wrong) expected
behavior: it previously asserted "A层"/"B层" should survive as entities
with INSTANCE-OF relations, which is exactly the leakage this fix
corrects.
…egory names

logic_rules/actions.linked_entities was saved straight from the raw LLM
output, unfiltered. Since the previous commit correctly stopped
extracting rule-category labels (e.g. "反欺诈规则") as entities, any
rule/action whose linked_entities still named one of those labels now
produced a dangling reference — confirmed on a real run: 101 P0
validation warnings on deepseek-v4-pro, 9 on deepseek-v4-flash, all
LOGIC_BROKEN_ENTITY_REF/ACTION_BROKEN_ENTITY_REF pointing at names that
are correctly absent from entities.

Filters both fields down to names present in entity_name_to_id (the
authoritative saved-entity set built earlier in the same function),
mirroring the dangling-relation cleanup already applied to `relations`
a few lines above.
…xtraction

graph-engineering playbook scaling guidance: a document large enough that
a single extraction call would have to hold the whole thing gets split at
markdown heading boundaries (falling back to paragraph boundaries for
headerless documents) into <=12000-char chunks, each carrying the tail of
the previous chunk as overlap so a concept mentioned right at a section
break isn't separated from the relations described just after it.

Reuses the existing cross-file _merge_extraction_results/resolve_entities
pipeline unchanged for cross-chunk merging — a chunk is just another unit
fed into the same executor loop that already handles multiple files, so
no new merge logic was needed.

Today's real documents (1.3-5.9KB) are all well under the 12000-char
threshold and pass through as a single chunk, unchanged from current
behavior; this is forward infrastructure for genuinely long documents.
…n stages

graph-engineering playbook model-selection guidance: extraction is high-
volume and schema-constrained (speed/cost dominate), while resolution and
relation inference weigh conflicting evidence (reasoning quality
dominates). Extraction always uses the model the user picked, unchanged.
For resolution and relation inference, _select_reasoning_model looks for
a non-fast sibling already configured on the same ModelConfig.models list
when the user's pick matches a known fast/cheap naming pattern (flash,
mini, haiku, lite, turbo, nano, fast) — no new user-facing setting, and
behavior is identical to today whenever no stronger sibling is
configured or the user already picked the strong variant.
graph-engineering playbook incremental-update guidance: resolve_entities
only clustered entities within the current run, so re-running extraction
on an ontology that already has saved entities could create a duplicate
for a surface-form variant of something that already exists (e.g. a new
run spells "客户" as "客户方").

resolve_entities gains an optional existing_entities param: entities
already saved on the ontology are added to each type's clustering
candidates, marked [已存在] in the prompt with an explicit instruction
that their name wins the canonical slot — backed by a deterministic
override so a re-run can never silently relabel an already-published
entity even if the model ignores the instruction. A lone new entity
(which previously skipped clustering entirely as "nothing to resolve
against") is now still checked when existing entities of its type exist.

run_extraction queries the ontology's current entities before resolution
and passes them through; no other behavior changes when no existing
entities are present (default param, backward compatible with the 3
pre-existing call sites).
graph-engineering playbook Section V.E: connected components, edge/node
density, and hub (highest-degree) entities are structural health signals
for whether entity resolution actually worked — a single component means
everything reachable; fragmented islands mean surface-form variants that
should have merged did not.

New app/services/graph_diagnostics.py: pure functions (no DB, no LLM
calls), computed over the ontology's FULL saved entity/relation set (not
just this run's output) so a stale count from a prior run never masks
today's actual state. Stored on extraction_task.validation_report under
a new graph_diagnostics key, alongside the existing P0 report — same
place, no new endpoint or table.

Isolated as its own module rather than folded into extraction.py: it's
pure graph theory unrelated to LLM orchestration, and reusable later for
an on-demand ontology-detail-page diagnostic outside the extraction
pipeline.
graph-engineering playbook Section V.A: an entity with enough graph
connectivity (degree >= 3) benefits from cross-relation synthesis — a
2-3 paragraph summary, 3-5 atomic traceable key facts, and a time range,
grounded in its own description and its known graph relations (no raw
document re-scanning needed; the description and relations are already
first-class extracted data). Stored under new summary/key_facts/
time_range/summary_degree keys in Entity.properties — description
itself stays untouched since several other places (resolution,
disambiguation) depend on it staying a short, one-line grounded string.

Re-summarizes only when degree changed since summary_degree was last
recorded, so a re-run doesn't re-call the model for an unchanged hub
every time. Uses the same reasoning_model_name already selected for
resolution/relation-inference — no new model-selection logic needed.

Fixes a real bug surfaced while testing the skip behavior: entity
upsert was replacing ent.properties outright on every re-run instead of
merging, silently destroying a previous run's summary (or any other
property a later run's extraction didn't re-emit).
Adds precision_recall() to the document-extraction eval harness, using
fuzzy (rapidfuzz) greedy one-to-one matching against a gold entity list
per case — a real quality signal instead of the existing loose
keyword-presence check. Wired additively into run.py (a domain without
a *.entities.json gold file simply skips it, no breakage).

Verified end-to-end with a live DeepSeek extraction run against real
test_data documents across 信贷/财务/供应链/医疗 (per explicit request
to cover all four, not just 信贷). Results (F1: 0.58/0.00/0.38/0.38)
show the harness correctly surfaces two distinct gap types: real
missed concepts (e.g. ACEI/ARB, 董事会, 审计委员会) and naming-
granularity mismatches where the extractor emits a more specific
compound name than the gold concept (财务's 0.0 case: 采购审批权限 vs
采购审批, 预算管理 vs 预算) — both are genuine signal, not tuned away.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkAraQUPvCkrAKpk89TMNR
app/database.py's SessionLocal is autoflush=False, but the graph-
diagnostics and hub-summarization block queried Entity/Relation back
from the session right after adding this run's own new rows, with no
db.flush() in between — so the query only saw whatever was committed
before this run started, missing everything just added.

Caught via a live Playwright-driven rebuild of the 信贷/财务/供应链/
医疗 simple-LLM ontologies: a brand-new 医疗 ontology's diagnostics
showed edge_count=0 despite 101 relations landing in the table
moments later. The existing unit tests never caught this because
their _worker fixture's session factory defaults to autoflush=True,
unlike production — added a dedicated regression test that builds its
session with autoflush=False to match app/database.py and reproduces
the bug (verified it fails without the db.flush() fix, passes with
it).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkAraQUPvCkrAKpk89TMNR
…pt/instance boundary

Adds an instance_relations output channel to the shared concept/instance
directive so a causal/interaction relation between two SPECIFIC named
things (e.g. 2型糖尿病 causes 糖尿病肾病, 阿司匹林 interacts_with 华法林)
survives even though relations only connect concept-level entities.
Reuses the existing but previously-unwired EntityInstanceRelation model
(an edge between two EntityInstance rows, anchored to the concept-level
Relation it instantiates) — no migration needed. Neo4j sync now also
pushes instance nodes and instance-relation edges, labeled distinctly
from their concept nodes.

Root cause of the reported bug: _merge_extraction_results rebuilds a
fresh dict from the per-chunk LLM outputs and was silently dropping any
key it didn't already know about, so instance_relations never survived
past the merge step even once emitted. Also needed an explicit db.flush()
between the auto-created concept Relation and the EntityInstanceRelation
that references it — autoflush's cross-table ordering wasn't reliable
here despite the declared FK, so plausible-looking automatic ordering
was made explicit instead of trusted.

Also updates the 医疗本体提取 prompt (DB-stored, not in git — this is a
manual content edit, not a code change) to drop a legacy "each document
must produce one type=Disease entity named after the filename"
instruction that predates the concept/instance discipline and directly
contradicted it — specific diseases/drugs now correctly go to instances
via entity_type, with the disease-drug relation graph preserved through
the new instance_relations channel instead of being silently discarded.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkAraQUPvCkrAKpk89TMNR
…uses them

Caught live on a real 供应链 rebuild: FK violation inserting into
entity_instance_relations because relation_definition_id pointed at a
Relation row that was still only pending in the session, not yet
flushed to Postgres.

Root cause: the earlier db.flush() fix only covered relations newly
created *within* the instance_relations loop itself. It missed the
case where a concept-level relation comes from the plain `relations`
array (the ordinary relations-upsert loop that runs just before) and
gets reused via relation_id_by_key by an instance_relations entry —
that relation was never flushed, so it doesn't exist yet when
EntityInstanceRelation's FK insert runs.

Also fixed the same test-environment gap as before: the two existing
instance_relations tests used the _worker fixture's default-autoflush
session, which silently flushes pending relations and therefore never
exercises this bug. Switched them (and the new regression test) to an
explicit autoflush=False session matching production's SessionLocal.

Also adds a domain-specific instance_relations example to the 财务,
供应链, and 通用本体提取 prompts (the last one is what 信贷 actually
runs under, since it has no dedicated prompt) — the medical prompt got
this treatment already; these three only had the shared directive's
single abstract example, which was clearly getting less attention in
their long, rule-heavy prompts than in medical's causal prose.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkAraQUPvCkrAKpk89TMNR
…delete

Two bugs reported by community feedback, both verified by direct
investigation before fixing:

1. An ontology with real instance data could never be deleted: the
   cleanup loop deleted agent_index_outbox before entity_instance_relations,
   but the latter's AFTER DELETE trigger re-emits fresh outbox rows —
   left uncleaned, they blocked the ontology row's own delete with
   ONTOLOGY_HAS_DEPENDENT_DATA. Reordered so agent_index_outbox is
   cleaned last, sweeping up anything the instance-table triggers just
   recreated.

2. Deleting an ontology or a curated dataset never touched Neo4j,
   Chroma, or MinIO, even though delete methods already existed for
   all three (Neo4jService.delete_by_ontology, ChromaService.
   delete_collection, StorageService.delete_object) — simply never
   called from any delete path, leaving permanent orphaned data.
   Ontology delete now best-effort cleans up its Neo4j nodes/relations
   and its `ontology_{id}` Chroma collection; curated dataset delete
   now best-effort removes each MediaItem's storage_uri/ocr_result_uri
   MinIO object. Both are wrapped so an unreachable external service
   never blocks or fails an already-committed Postgres delete.

Regression tests: a live-Postgres test reproduces the outbox trigger
bug end to end (fails on the old cleanup order, passes on the new
one); mocked-service tests confirm the external cleanup calls fire
with the right ontology/collection id and that failures there don't
prevent the 204 response.

Two other reported issues (permanently-immutable published-ontology
releases, and the resulting misleading delete error message) are a
product/compliance decision about audit-immutability boundaries, not
addressed here. A third (legacy v2_curated_datasets table not cleaned
up by curated delete) needs its own migration-status investigation,
also not addressed here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkAraQUPvCkrAKpk89TMNR
…nection test

Reported by a user: testing a Pipeline Mapping data-source connection
always showed "success" regardless of host/port/username/password.

The backend (app/routers/v2/connections.py's /connections/test-config)
was never the problem — it genuinely attempts the connection and
returns HTTP 200 with {"success": false, "detail": "..."} on failure
rather than raising, so wrong credentials or an unreachable host still
come back as a normal 2xx response. The frontend's test-connection
button awaited that response and, since axios only rejects on non-2xx,
treated any resolved promise as success without ever looking at the
`success` field in the body — so the only way to see "failed" was a
network error or an auth/validation rejection, never an actual bad
connection.

Now the button reads `result.success` and falls back to the failed
state (using the backend's `detail` message) when it's false.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkAraQUPvCkrAKpk89TMNR
Reported gap (#5 from earlier community feedback triage): deleting a
curated dataset left behind a permanent orphan once it had ever gone
through review.

_ensure_curated_dataset_row (and the older /review endpoint's inline
equivalent) lazily creates a v2_curated_datasets row sharing the SAME
id as the real v2_datasets row, purely as an FK bridge so CuratedReview
has something to point at — nothing else links the two tables.
delete_curated only ever deleted the v2_datasets row and the
CuratedReview rows filtered by that shared id; the CuratedDataset
mirror row itself was never targeted by any delete path, so a
reviewed-then-deleted dataset always left one behind.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkAraQUPvCkrAKpk89TMNR
…sible fixes

Reported gap (#3 from earlier community feedback triage): the two
ontology-delete failure messages told users to do something that no
mechanism in the app can actually do.

ONTOLOGY_BOUND_TO_AGENT said "unbind the agent first" — no unbind
capability exists anywhere (backend or frontend); the only code that
ever deletes an agent_ontology_bindings row is the agent's full
hard-delete. Building a real unbind would mean letting a historical
agent version's ontology binding be removed, which conflicts with
this codebase's explicit agent-version immutability guarantee ("old
versions stay byte-identical") — the same category of decision as the
release-immutability question (#2), deliberately deferred rather than
solved here.

ONTOLOGY_HAS_DEPENDENT_DATA said "clean up first" — the dependent data
is exactly the execution history rooted through ontology_releases,
which a DB trigger (reject_ontology_release_mutation) makes
unconditionally immutable with zero exception path.

Both messages now state the actual situation instead of pointing at a
remedy that doesn't exist, in both zh and en.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AkAraQUPvCkrAKpk89TMNR
…outes

Legacy v1 ontology routers (entities, logic, actions, graph, files, export,
audit, extraction, ontology get/update/delete) and the classic v2 graph,
search and logic/action routers only required a logged-in user. Any
account, including a self-registered viewer, could list ontology ids via
/overview/stats and read or export every ontology; editors could modify
ontologies they had no grant on.

Every route now checks the caller's ontology_project_access_grants row
(read or edit) and answers 404 ONTOLOGY_NOT_FOUND when it is missing, so
the existence of other ontologies is not disclosed. Overview stats only
count discoverable ontologies, and Neo4j traversals stay within the
ontology they were asked about.

Ontologies inserted by direct SQL have no creator grant; run
scripts/backfill_creator_grants.py after deploying.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
File and dataset uploads read the whole body into memory before checking
max_upload_mb, so one oversized request could exhaust the worker. Both
now read in 1 MiB chunks and stop with 413 as soon as the limit is
crossed. A rejected or failed upload no longer leaves a partial file on
disk, and deleting a file removes it from disk only after the row
deletion commits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ources

Pipeline connector nodes and refresh runs now resolve MySQL/Postgres
sources from a saved, encrypted Connection instead of inline credentials:

- connection/config.py is the single place that decrypts a Connection and
  builds the SQLAlchemy URL; the router, pipeline runs and refresh polling
  all go through it. Connection test errors are passed through
  safe_connection_error so credentials don't leak into responses.
- New GET /connections/{id}/resources and .../resources/{resource}/sample
  let the Builder list tables and preview rows.
- Pipeline definitions drop inline SQL credentials on save and hide them
  on read. Inline Mongo/REST connector nodes are rejected with
  UNSUPPORTED_INLINE_CONNECTOR.
- A pipeline run snapshots each selected table as an immutable
  DatasetVersion and pins every input version before pulling.
- Refresh runs resolve a concrete resource instead of the "__scheduled__"
  placeholder. Pipeline-target schedules are closed, and legacy rows are
  disabled at dispatch.
- Frontend: ConnectorInspector selects a saved connection and table, and
  ConnectionsTab can test connections.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on_id

delete_model always cleared model_configs.active_version_id, a column
that only exists after the versioning migration. On a schema built by
metadata.create_all that UPDATE failed. Skip it when the versioning
schema isn't present; migrated databases keep their reference guards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Declare requires-python ">=3.11,<3.13" and make the startup guard reject
3.13+ as well as <3.11, with a message that names both bounds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Inside the compose network Chroma listens on its container port 8000;
8001 is only the host-side mapping. Set CHROMA_PORT=8000 for the backend
and worker services and in .env.example. README now lists React 19.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FIXED_NOW was captured when the module was imported, so in a long run the
fixtures could drift into a different lag bucket by the time a test ran.
Each fixture and test now reads the clock itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
react-hooks/set-state-in-effect flagged resets inside effects in
ConversationPanel, V2DefinitionEditor and the four runtime pages. Per-id
state is now keyed by the id it belongs to, or the content component is
remounted with a key, so the effects only start loads. Effects also
ignore results that arrive after a newer request, and the tests cover
switching ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The API client cleared the session on any 403, and the "Publish All"
buttons in the Logic and Actions tabs call endpoints that always return
403 PUBLICATION_NOT_ENABLED (publication goes through the ontology
publish flow), so clicking them sent the user back to the login page.

- Only 401 (after a failed token refresh) logs the user out; a 403 is a
  permission error, not an expired session.
- Remove the two "Publish All" buttons and their orphaned i18n keys.
- Hide the admin-only curated approve/reject buttons from non-admins,
  which used to hit the same 403 path.

Closes #37

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OntologyPublicationPanel shows "publish new version" only when the
ontology is published and isDirty is true, but the API never returned
ontology_projects.is_dirty and the detail page never passed it, so the
button could not appear after editing a published ontology.

OntologyOut now includes is_dirty and the detail page passes it to the
panel. The page refetches the ontology when tab mutations settle and
after lifecycle actions, so the flag updates without a reload.

Closes #38

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Binding an ontology to an agent requires the read_schema, read_instances
and traverse_relations data capabilities whatever tools are selected.
These come only from an admin-issued ontology data grant; creating an
ontology doesn't grant them. Users without a grant saw a bare "binding
conflict" plus an ontology id, and clearing tools changed nothing.

- The banner now names the blocked ontology and says to ask an admin
  for a data grant.
- Clear the validation result when the binding set is empty, so an old
  conflict banner doesn't stay after unbinding.

The access model is unchanged: owners still need an explicit data grant.

Closes #39

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jingw2
jingw2 merged commit 6f1b247 into master Nov 13, 2026
1 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant