Repository navigation
feat(comprehension): ingestion-time LLM digests so ask/chat answer structure + combination questions - #1158
Merged
Conversation
Dual-index retrieval, CRE summaries, prior cage, hub seed, edition remap, and related librarian defaults. Rewrites #1088 as the reviewable base; later stacked PRs add harvester, B2 eval, fixtures, mappings, and full-pipeline docs.
- Refuse plain-text fetch redirects that leave GitHub raw hosts (SSRF). - Re-check expected embedding dimension on query-cache hits. - Fill phrase-only oie blocks in apply_oie_fill_if_missing without losing phrases. - Expose last_preferred_cre_ids on StandardHopRetriever and honour its allowlist. - Commit each metadata enrichment so a later failure cannot roll back earlier work. - Load .env from the repo root in the enrichment scripts. - Parse hostnames for CRE hub sync SSL and loopback checks; batch SQLite IN lookups; stop echoing database credentials.
Optional requirement extract (off/auto/on) plus GitHub tarball/URL ingest wiring for Module B → C. Stacked on #1088 librarian base.
Harness runners, hop-distance scoring, and gold loaders without bulky source fixtures (those land in follow-up stacked PRs).
Wire LlmSafetyGuard (LiteLLM) as the live factory default so runs evaluate safety instead of NullSafetyGuard. Apply the guard on every decision path, prefer safety reason codes over empty shortlists, and cover a synthetic adversarial corpus in unit tests.
- Do not fall back to main/master when the user named a branch. - Download tarballs with a timeout and a size cap; cap per-member and total markdown bytes read from archives. - Validate owner, repo and branch and URL-encode them for codeload. - Delete prior ingest rows only after download and chunking succeed, and never rows from other runs. - Pass the model override through a local config instead of os.environ.
- Redact database passwords in orchestrator, B2 and setup_oie output; write tmp/oie.env with mode 600. - Keep the existing --db for --skip-a and a user-supplied --repos-yaml. - Create the artifact dir before writing orchestrator reports. - Clear per-model knowledge rows between Qwen and Gemini classify runs. - Key the switch-grid embed cache by model, honour --force with stale-lock recovery, write explicit zero flags to stack_winners.yaml, and tolerate LibrarianConfig fields that are env-only. - Match B1 sheet-grain gold exactly instead of by substring.
Large fixture slice for ASVS eval arms only — split from NIST/other sources so reviewers can treat this as mechanical data review.
NIST OSCAL sample plus Top10/API/K8s/LLM/AISVS/agentic fixture sources and gold JSON. Stacked after the ASVS-only fixture PR.
Update OWASP mapping fixtures and loader helpers so orphan umbrella Related targets become canonical gold for eval and product mappings.
- Forward leaf_drilldown_min_sections=0 instead of replacing it with 20. - Make queue wiping in run_full_pipeline opt-in via --wipe-queues and add the flag to the detached clone-DB runner, which now reports its real exit code. - Fall back to sys.executable when venv/bin/python is missing. - Fix the B2 report gate string (top-3) and drop the nonexistent make target from the docs. - Make the denied-resource drill-down test actually reach the allowlist gate.
Prior-caged retrieval embeds the same header and body several times per chunk. Prefetch the unique strings for a queue batch so those Gemini calls collapse into one list request.
Makefile/full-pipeline glue, OIE human docs, and experiment YAMLs. Experimental flags are documented as removable on merge if they are not useful to keep.
Introduce a feature-flagged Nest+GitHub hybrid index and chat router so community/meta questions (chapters, projects, events, board history, geo-nearest meetups) are answered fail-closed without routing through CRE RAG. Includes strict auto-concepts, merge cron target, sync Makefile targets, and offline demo-scenario tests. Co-authored-by: Spyros <northdpole@users.noreply.github.com>
Six-lever ablations selected SHORTLIST_JUDGE=1, UMBRELLA_PROMOTE_CAP=8, and GitHub/B2 score top_k=3 (KEEP_HUB/HUB_FIRST off). Wire those as pipeline defaults with leaf-drilldown gating and the d1 lever experiment harness.
Keep Nest/GitHub concepts out of the CRE graph and citation table. Chat returns empty CRE references and external owasp_agent citations; the chatbot UI never links meta hits into OpenCRE nodes. MCP exposes local owasp_meta_* tools (flag-gated) that read the separate index. Co-authored-by: Spyros <northdpole@users.noreply.github.com>
Creates meta_entity, owasp_concept, sync_log, and concept_merge_audit when missing so IndexStore.create_all environments stay upgrade-safe.
…facts. Route hard probes correctly (inactive chapters, geo neighbors, leaders, membership pricing, fail-closed talks, CRE-844-486) and extend the index/crawler so gold+synth evals pass.
Vendors the 400 unique offline probes plus router/slot fixes that lift agent-path scoring to pass_rate>=0.90 with zero hallucinated talks.
Adds a package README (keys, sync, chat/MCP, suite scoring), a public fixture scorer script, and a Makefile target so contributors can reproduce agent-path eval without the private homeserver harness.
Ingest project leaders from leaders.json, add person/project/compare routing, and ground AI Exchange lethal-trifecta answers so Rob/AIX/GenAI questions stop falling through to CRE RAG hallucinations.
Pass a capped messages[] history through /completion into LiteLLM and the OWASP agent so clarify follow-ups keep prior context; UI gets a New chat control that clears the local thread.
Prefer main DATABASE_URL for the agent index, fail-closed talk+CRE compounds, board-person without year prompts, and geocode fallbacks used on staging.
Replace brand/AIX keyword classifiers with cre_graph / corpus / owasp_agent_meta scoring; embed agent meta (+ evidence notes) into the shared embeddings table behind source_channel.
Use Gemini REST by default with batched commits so litellm/Flask no longer OOM-kills the backfill.
Store gemini/gemini-embedding-001 so meta rows match cre_graph/corpus and the completion embedding contract check passes.
…EST_EVENTS and OIE_EMBED_FILED_NODES Embedding auto-filed nodes spends provider tokens and writes the embeddings table, so it is opt-in (OIE_EMBED_FILED_NODES=1) like OIE_HARVEST_EVENTS.
* trim/1136: fix(oie-scheduler): embed_filed_nodes defaults off, document OIE_HARVEST_EVENTS and OIE_EMBED_FILED_NODES fix(owasp-agent): admin sync and scheduler stage apply the opt-in meta-embedding push chore(owasp-agent): type embed_sync and its test chore: drop generated frontend bundle (rebuild at deploy) refactor(owasp-agent): move the meta-embedding push out of sync_all into embed_sync # Conflicts: # application/frontend/www/bundle.js # application/tests/admin_panel_test.py # docs/oie/feature-flags.md
* trim/1142: fix(oie-scheduler): embed_filed_nodes defaults off, document OIE_HARVEST_EVENTS and OIE_EMBED_FILED_NODES fix(owasp-agent): admin sync and scheduler stage apply the opt-in meta-embedding push chore(owasp-agent): type embed_sync and its test chore: drop generated frontend bundle (rebuild at deploy) refactor(owasp-agent): move the meta-embedding push out of sync_all into embed_sync
* trim/1155: fix(oie-scheduler): embed_filed_nodes defaults off, document OIE_HARVEST_EVENTS and OIE_EMBED_FILED_NODES fix(owasp-agent): admin sync and scheduler stage apply the opt-in meta-embedding push chore(owasp-agent): type embed_sync and its test chore: drop generated frontend bundle (rebuild at deploy) refactor(owasp-agent): move the meta-embedding push out of sync_all into embed_sync
* trim/1159: fix(oie-scheduler): embed_filed_nodes defaults off, document OIE_HARVEST_EVENTS and OIE_EMBED_FILED_NODES fix(owasp-agent): admin sync and scheduler stage apply the opt-in meta-embedding push chore(owasp-agent): type embed_sync and its test chore: drop generated frontend bundle (rebuild at deploy) refactor(owasp-agent): move the meta-embedding push out of sync_all into embed_sync
…e code default The example enabled long-context reading with a 2,000,000 character cap (about 500k tokens per question on public chat). Match the code: off, 700000.
* trim/1127: test(owasp-agent): sync and concepts must not import application.database
* trim/1137: test(owasp-agent): sync and concepts must not import application.database
* trim/1136: test(owasp-agent): sync and concepts must not import application.database
…and counts real vectors
* trim/1142: fix(admin): CRE embedding guard only treats a missing table as empty and counts real vectors test(owasp-agent): sync and concepts must not import application.database
* trim/1155: fix(admin): CRE embedding guard only treats a missing table as empty and counts real vectors test(owasp-agent): sync and concepts must not import application.database
* trim/1159: fix(admin): CRE embedding guard only treats a missing table as empty and counts real vectors test(owasp-agent): sync and concepts must not import application.database
…d of patch.dict in the embed_sync tests
* trim/1127: test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests
…embed_fn instead of patching embed_sync
* trim/1137: test(owasp-agent): exercise the meta-embedding push through injected embed_fn instead of patching embed_sync test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests # Conflicts: # application/tests/oie_scheduler/runner_test.py # application/utils/oie_scheduler/jobs.py
…he new opt-in assertions
* trim/1136: test(oie-scheduler): restore the environment with a plain helper in the new opt-in assertions test(owasp-agent): exercise the meta-embedding push through injected embed_fn instead of patching embed_sync test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests # Conflicts: # application/tests/admin_panel_test.py # application/utils/admin_panel/service.py
…vironment with a plain helper in the #1142 tests
* trim/1142: docs(staging): use placeholder tester addresses; test: restore the environment with a plain helper in the #1142 tests test(oie-scheduler): restore the environment with a plain helper in the new opt-in assertions test(owasp-agent): exercise the meta-embedding push through injected embed_fn instead of patching embed_sync test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests
* trim/1155: docs(staging): use placeholder tester addresses; test: restore the environment with a plain helper in the #1142 tests test(oie-scheduler): restore the environment with a plain helper in the new opt-in assertions test(owasp-agent): exercise the meta-embedding push through injected embed_fn instead of patching embed_sync test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests
* trim/1159: test(owasp-agent): restore the environment with a plain helper in index_routing_test docs(staging): use placeholder tester addresses; test: restore the environment with a plain helper in the #1142 tests test(oie-scheduler): restore the environment with a plain helper in the new opt-in assertions test(owasp-agent): exercise the meta-embedding push through injected embed_fn instead of patching embed_sync test(owasp-agent): restore the environment with a plain helper instead of patch.dict in the embed_sync tests
northdpole
added a commit
that referenced
this pull request
Nov 1, 2026
* trim/1158: trim(#1158): strip_injection no longer truncates legitimate questions; correct default-off claims trim(#1159): drop cambridge->boston alias, dead locals, test env/tempfile leaks trim(#1155): drop laptop-local edge probe scripts, make node metadata seeding opt-in, fix empty-key blurb match trim(#1142): admin upstream-sync-and-ingest no longer replaces existing CRE embeddings trim(#1142): no credentialed CORS without an allowlist, OIE workers opt-in in import-all, guard levers-matrix queue reset trim(#1137): drop brevity-bot skills and stale handover doc, restore README/dev setup, loopback guard for run_admin_local.sh trim(#1127): drop adversarial asker/batch scripts and canned CRE follow-ups, set source_channel in raw embedding writers trim(#1120): fix merge-conflict markers, guard full pipeline, drop dead experiments style: black-format llm_safety_guard.py chore(oie): trim #1118 for the staging rollout chore(oie): trim #1117 for the staging rollout chore(oie): trim #1116 and guard destructive eval scripts chore(harvester): trim #1115 for the staging rollout chore(oie): trim #1088 for the staging rollout test: embedding-failure scheduler test needs graph filing enabled now that it defaults off style: format files flagged by CI (black, prettier) Make event-repo harvesting opt-in (OIE_HARVEST_EVENTS) Make event page parsing linear-time and clean Playwright fallback text Embed auto-filed graph nodes in the CRE expansion job Parse OWASP event pages and harvest event repos # Conflicts: # application/mcp/ask.py # application/web/web_main.py
northdpole
added a commit
that referenced
this pull request
Nov 1, 2026
* trim/1162: (27 commits) docs(env): document CHAT_BLOCK_UNVERIFIED (default on, changes public chat answers) trim(#1162): drop prune_noise_chunks (unguarded DELETE on embeddings), validate authority weights before use, document ANSWER_LOG_*, no host paths in sync_check_staging trim(#1158): strip_injection no longer truncates legitimate questions; correct default-off claims trim(#1159): drop cambridge->boston alias, dead locals, test env/tempfile leaks trim(#1155): drop laptop-local edge probe scripts, make node metadata seeding opt-in, fix empty-key blurb match trim(#1142): admin upstream-sync-and-ingest no longer replaces existing CRE embeddings trim(#1142): no credentialed CORS without an allowlist, OIE workers opt-in in import-all, guard levers-matrix queue reset trim(#1137): drop brevity-bot skills and stale handover doc, restore README/dev setup, loopback guard for run_admin_local.sh trim(#1127): drop adversarial asker/batch scripts and canned CRE follow-ups, set source_channel in raw embedding writers trim(#1120): fix merge-conflict markers, guard full pipeline, drop dead experiments style: black-format llm_safety_guard.py chore(oie): trim #1118 for the staging rollout chore(oie): trim #1117 for the staging rollout chore(oie): trim #1116 and guard destructive eval scripts chore(harvester): trim #1115 for the staging rollout chore(oie): trim #1088 for the staging rollout fix(auth): harden web sign-in found by adversarial review fix(auth): /auth/user needs any role, not admin fix(auth): carry the PKCE code_verifier from login to callback fix(auth): build a fresh OAuth flow per login instead of a shared singleton ...
northdpole
changed the base branch from
claude/fix-agent-index-80b1a2
to
staging
November 1, 2026 22:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1155.
Why
#1155 answered structural questions (e.g. SAMM levels) from hand-written facts, which cannot scale and was partly wrong. This derives that knowledge at ingestion instead.
How
application/utils/comprehension/: reads whole harvested documents, extracts claims that must include a verbatim quote found in the source (unverified claims dropped), builds a per-standard overview that must cite existing documents, and stores everything as embedded evidence docs in the existing agent channel.UNKNOWNfalls through to existing paths. Flag:OWASP_COMPREHENSION_ENABLED(default on).scripts/comprehend_standard.py --repo OWASP/SAMMruns it per harvested repo.docs/oie/comprehension-design.md.Verified on staging (homeserver)
SAMM/ASVS/MASVS/AI guide digested; e.g. "how many levels in owasp samm" returns three maturity levels per practice (also surfaces a v1.5 vs v2 conflict, to fix by preferring the latest version), with GitHub citations.
Not done yet
Generated source-grounded eval (Gemini questions, Claude judge), held-out standard test, version preference, removal of hand-written
node_facts, people/talks layer, scheduling into the pipeline.🤖 Generated with Claude Code