From e0406a9e4b4f000f6829e73470e2bf07bfcabbed Mon Sep 17 00:00:00 2001 From: zhenkun Date: Fri, 2 Oct 2026 02:23:28 -0400 Subject: [PATCH 1/5] docs: plan Insight refactoring and Atmos-RAG integration --- REFACTOR_INTEGRATION_PLAN.md | 445 ++++++++++++++++++ docs/REFACTOR_STATUS.md | 72 +++ .../2026-10-02-refactor-plan-consistency.md | 75 +++ 3 files changed, 592 insertions(+) create mode 100644 REFACTOR_INTEGRATION_PLAN.md create mode 100644 docs/REFACTOR_STATUS.md create mode 100644 docs/reviews/2026-10-02-refactor-plan-consistency.md diff --git a/REFACTOR_INTEGRATION_PLAN.md b/REFACTOR_INTEGRATION_PLAN.md new file mode 100644 index 0000000..5f5a945 --- /dev/null +++ b/REFACTOR_INTEGRATION_PLAN.md @@ -0,0 +1,445 @@ +# Insight Refactoring and Atmos-RAG Integration Plan + +Date: 2026-10-02, America/New_York\ +Plan revision: 1\ +Product standard: a maintainable, single-user, local-first application\ +Delivery method: one bounded, verifiable increment at a time + +This plan makes Insight the application and engineering home. Atmos-RAG contributes its source organization, meteorological evidence requirements, and experimental method. Refactoring existing behavior takes priority over adding features or replacing the stack. + +The current request authorizes this planning package. It does not start the implementation increments below. Current authorization, progress, blockers, and the resume point belong to [REFACTOR_STATUS.md](docs/REFACTOR_STATUS.md). The planning consistency review is [recorded separately](docs/reviews/2026-10-02-refactor-plan-consistency.md). + +## 1. Intended outcome and engineering standard + +The application should let its owner import selected local material, search within an explicit scope, inspect original evidence, and obtain an answer whose citations can be checked. It must explain missing evidence and unavailable dependencies without reporting them as successful answers. + +Success means: + +- The owner can understand the important modules, run the application from documented steps, and diagnose failures. +- Keyword search remains usable without a vector service or a generating model. Answer generation reports its actual readiness separately. +- Search, chat, and streaming chat honor the same eligible-document rules and requested scope. +- Document changes, failed indexing, retries, restart recovery, and logical removal keep the visible evidence set consistent. +- A citation resolves to a specific source revision and location; a candidate score is not presented as answer correctness. +- Application data and source material remain recoverable. Migration and rollback procedures are verified on isolated copies before actual migration. +- Tests explain behavior and failure recovery. A small demonstration dataset is not advertised as professional validation. +- A later session can resume from repository records without depending on chat history. + +Maintainability does not mean introducing microservices, a framework for every operation, or arbitrary coverage and file-count targets. Use consequences, observed failures, module responsibilities, and recovery needs to determine scope. + +## 2. Scope and explicit exclusions + +| Area | Planned baseline | Outside the current plan's default implementation | +| --- | --- | --- | +| Application home | Existing Insight repository and Python application | A second maintained backend in Atmos-RAG; repository renaming | +| User and platform | One local user; macOS is the first acceptance environment | Public hosting, multi-user tenancy, enterprise RBAC, universal platform support | +| Input | Selected PDF, Markdown, and TXT files; explicit local source roots | Scanning the entire home directory or Obsidian vault | +| Retrieval | Searchable original material, lexical/BM25 baseline, optional existing vector path | Mandatory hybrid retrieval or mandatory reranking | +| Evidence | Source revisions, locations, relevant sections, validated citation identifiers | A promise of automatic scientific truth verification | +| Integration | One-way reading of selected Atmos-RAG/Obsidian material | Bidirectional synchronization or automatic edits to source notes | +| Runtime | Local model adapter and local storage; explicit degraded capabilities | Unapproved cloud calls, model training, Kubernetes, managed services | +| Later experiments | Only extensions supported by labeled failure cases | GraphRAG, autonomous agent loops, visual retrieval, fine-tuning, scientific array computation by default | + +The current Atmos-RAG archive and Obsidian originals remain preserved. Integration does not require moving their master files into Git, copying every PDF, or deleting old folders. A missing source becomes unavailable in the application's catalog; filesystem cleanup remains a user action under the current no-deletion instruction. + +## 3. Baseline and evidence limits + +Baseline application commit: `abb59d7515119e9dcc4ee13bc307cbdc4bed8841`. + +The 2026-10-02 investigation found: + +- Insight contains the application, tests, evaluation script, browser console, and eleven main OpenSpec capability directories. +- Atmos-RAG has no local Git repository or application implementation. Its archive registers a separate Obsidian knowledge directory. +- The Obsidian materials include thirteen Markdown notes, seven RAG-methodology PDFs, and twelve unannotated question seeds. The meteorology directory contains its usage note, not an established domain corpus. +- Insight's evaluation uses three sample documents and twelve questions, including two refusal questions. These are smoke-test assets. +- Existing tracked Insight contents match the baseline; 144 file-mode differences existed before this task and must remain outside its commits. +- Baseline GitHub CI succeeded. Prior controlled checks passed syntax/lint and fifteen existing pure-memory test functions, but did not constitute a full local pytest run or real Ollama/Milvus qualification. +- Current OpenSpec CLI and the ordinary local Python dependency environment are unavailable. Recheck them in R01; do not invent an installation or validation success. + +These are dated observations, not future acceptance receipts. Prior conversation probes identify work; each implementation increment must reproduce relevant failures and verify its final content. The actual result of this planning package is recorded in the review and status files. + +Baseline references: + +- [Application composition](app/main.py), [configuration](app/core/config.py), [API](app/api/routes.py), and [API models](app/schemas/api.py). +- [Ingestion](app/services/ingestion.py), [tasks](app/services/jobs.py), and [catalog](app/services/catalog.py). +- [Hybrid retrieval](app/retrieval/hybrid.py), [vector adapter](app/retrieval/vector.py), [BM25](app/retrieval/bm25.py), and [provider protocols](app/retrieval/protocols.py). +- [Parsing](app/ingestion/parsers.py), [chunking](app/ingestion/chunker.py), [QA](app/services/qa.py), and [browser console](app/web/app.js). +- [Evaluation](scripts/evaluate.py), [tests](tests), [dependencies](pyproject.toml), [lockfile](uv.lock), [CI](.github/workflows/ci.yml), and [deployment](docker-compose.yml). + +Machine-specific source roots belong to local configuration, using names such as `INSIGHT_ROOT`, `ATMOS_ARCHIVE_ROOT`, and `ATMOS_KNOWLEDGE_ROOT`. Do not publish private paths, notes, credentials, or raw runtime records in planning documents. + +## 4. Decision ownership and document responsibilities + +| Record | Owns | Does not own | +| --- | --- | --- | +| This plan | Product boundary, proposed architecture, increment sequence, workflow and acceptance rules | Live completion state, implementation authorization, test results | +| [Status](docs/REFACTOR_STATUS.md) | Current scope, active increment, authorization reference, blockers, next action, links to decisions and receipts | Independent grants of permission or copied specifications | +| [Main OpenSpec specifications](openspec/specs) | Current documented behavior contracts | Proof that implementation meets the contracts | +| An implementation change's OpenSpec artifacts | Its proposal, design, delta requirements, and tasks | Authority to push, merge, migrate private sources, or advance another increment | +| Git | Actual file content, branches and commit history | Professional acceptance or user permission | +| Increment verification receipt | Commands, artifact/version identity, outcomes, limitations, acceptance evidence | Permission or a replacement task list | +| User messages | Task-specific authorization and acceptance | Automatic permission inferred from generated documents | + +Read live code and Git to verify state. If the plan, status, specifications, and implementation disagree, record the discrepancy and reconcile the affected scope before claiming completion. Do not silently choose whichever record is most convenient. + +Planning constraints already established by the user: + +- Insight is the application home; prioritize refactoring and personal use. +- Work one increment at a time; retain existing source material and unrelated changes. +- Follow the no-filesystem-deletion rule, including test cleanup and temporary artifacts. +- New dependencies require permission. Installing already-declared dependencies can be routine within an authorized increment, after reviewing installation side effects. +- Commit completed task changes after appropriate verification; push requires explicit authorization for the current branch and destination. Merge and release are separate. + +Proposed engineering choices, to be resolved in the affected increment's design: + +- Preserve a modular monolith, SQLite, and a lexical baseline. +- Keep original sources external and read them one way. +- Publish validated index generations rather than mutating the active evidence set during construction. +- Separate stable document identity, content revision, evidence location, and encoding/index configuration. +- Return per-request retrieval reports and use catalog-backed evidence hydration. + +Creating this plan does not make every proposed design decision accepted. Record later accepted or superseded decisions in the decision section of the current change's design, with their reason and user reference where authorization matters. Status stores a pointer, not another copy. + +## 5. Target responsibilities and contracts + +Keep existing directories unless responsibilities justify a move. The table describes responsibilities, not a mandate to create every named class or a large new directory tree. + +| Responsibility | Existing home | Target contract | +| --- | --- | --- | +| Composition and configuration | `app/main.py`, `app/core` | Explicit startup/shutdown; validated configuration loaded at construction; no hidden resource creation in library imports | +| Source intake | `app/services/ingestion.py`, future source adapter | Upload or allowed-root read; input receipt; source provenance; no modification of master material | +| Parsing and evidence structure | `app/ingestion` | Original-to-derived location mapping, title hierarchy, paragraph/section relationships, explicit unsupported or partial content | +| Catalog and tasks | `app/services/catalog.py`, `jobs.py` | Stable identities, revisions, states, job lineage, reproducible build configuration, recoverable publication | +| Retrieval | `app/retrieval` | One scope contract; per-request results, status, timings and score meaning; backend-independent evidence identifiers | +| Evidence reading | Initially a focused service alongside QA | Hydrate identifiers from the catalog; recover relevant parent sections under a context budget | +| QA and sessions | `qa.py`, `session.py` | Separate conversation aid from evidence; citation validation; explicit success/refusal/partial/failure states | +| Provider adapters | `ollama.py`, `rerank.py`, `vector.py` | Query/document encoding semantics, batch support, versions, timeouts, readiness and failure categories | +| API and browser | `app/api`, `schemas`, `web` | Compatible public behavior, explicit scope, inspectable source location, accurate progress and terminal states | +| Evaluation | `scripts/evaluate.py`, `tests` | Shared production configuration, annotated evidence, meaningful denominators, isolated deterministic and live checks | + +### 5.1 Scope and retrieval report + +Scope includes source/document selection, domain, revision eligibility, and logical removal/index status. Existing filters should map to it without silently broadening a request. An explicitly empty allowed set means no evidence, not unrestricted retrieval. + +Eligibility refers to a validated active revision and the capability requested. A failed staged replacement or optional vector build does not invalidate a still-usable lexical generation. Retained active content can remain readable while its replacement is pending, with that condition visible; superseded, logically removed or unavailable content cannot silently re-enter current results. Define any explicit snapshot-reading policy separately from current-source search. + +The retrieval operation returns its own results and execution report. Eliminate consumers' dependence on mutable `last_status`/`last_timings`. Keep raw lexical, vector, fusion, and rerank scores distinct; record their meaning. A ranking score is not a calibrated answer probability. + +Backends should filter before candidate truncation where supported. Where a backend cannot honor a scope correctly, report the limitation or use a correct scoped fallback; do not treat fixed global overfetch as complete scoped retrieval. + +### 5.2 Source identity and index lifecycle + +Use distinct concepts for stable document identity, immutable content revision/hash, source locator, derived evidence unit, and build configuration. Preserve legacy IDs through an explicit mapping when migrating. Only add persisted fields that have a demonstrated consumer or integrity requirement. + +The publication protocol must define ordering across catalog, lexical index and optional vectors. SQLite and vector storage do not share a transaction. Construct and validate a new generation, then switch the active catalog reference only when required outputs are ready. Retain the prior usable generation and define restart behavior for every interrupted step. + +Keyword-only operation must remain a legitimate configuration. If vectors are optional and their build fails, record separate capability readiness; do not misreport the vector build as successful or expose an ambiguous partially published state. + +A new file revision remains staged until ready. Source edits, logical removal, failed builds and model changes must have deterministic consequences for current retrieval and old citations. Old citations remain tied to their revision; if retained source content is unavailable, show that limitation instead of redirecting to a newer revision. + +### 5.3 Evidence and answers + +Retrieval units may be smaller than the evidence supplied to generation. Hydrate the relevant section, variable definition and conditions where available. Apply a recorded context budget; label omissions instead of implying complete reading. + +Citation identifiers must exist in the supplied evidence set. Display retrieved candidates separately from actually cited sources. Deterministic citation validation is necessary but does not prove that a scientific claim is entailed. Domain usefulness remains a human acceptance item. + +Keep streaming and non-streaming semantics aligned. Confirm successful stream termination, preserve partial output on failure, and make the browser distinguish refusal, model failure, partial output, fallback and owner cancellation. Verify cancellation propagation with a real provider before claiming it stops model computation. + +## 6. Traceability from findings to increments + +The IDs below identify work, not independently verified defects in every environment. R01 refreshes evidence classifications; later receipts close them with final-state checks. + +| Finding | Refactoring concern | Owner | Required behavior evidence | +| --- | --- | --- | --- | +| F01 | Search/chat scope mismatch and invalid-source eligibility | R02 | Search, chat and stream exclude the same out-of-scope, unready, inactive, unavailable and logically removed revisions under the requested capability | +| F02 | Shared retrieval report races | R02 | Overlapping successful/failed requests retain their own results, errors and timings | +| F03 | Vector post-filter truncation misses scoped evidence | R02 | A valid scoped hit outside the global candidate window is still retrievable | +| F04 | Offset pagination limited by fixed candidate count | R02 | Paging policy is explicit and stable beyond the first candidate window | +| F05 | Failed duplicates reported as successful | R03 | Same-content failed/stale builds execute recovery; successful equivalent builds remain idempotent | +| F06 | Retry input/lineage and queued restart recovery gaps | R03 | Retry has durable input and parent linkage; restart accounts for queued/running work | +| F07 | Stale lexical contents and inconsistent publication | R03 | Failure/logical removal cannot leak invalid evidence; interrupted publication retains a usable state | +| F08 | Configuration/import/lifecycle and resource limits | R01, R03 | Documented config works; imports are inert; upload/queue/index resources have tested bounds | +| F09 | Source metadata loss and weak version tracking | R04 | Every retrieval path resolves the same revision, page/section and original source | +| F10 | Obsidian exclusions/frontmatter not enforced | R06 | Denied directories, `rag_index: false`, drafts and evaluation content remain excluded | +| F11 | Title hierarchy and evidence context loss | R05 | Title-only terminology is searchable; formulas and conditions can be read together | +| F12 | Invalid/unused citations accepted | R05 | Invalid citation IDs cannot be reported as a validated answer; used sources are distinguishable | +| F13 | Ranking used as evidence sufficiency | R05 | Lexical topic overlap without answer evidence is handled explicitly; score meanings remain distinct | +| F14 | Stream, UI readiness and session behavior mismatch | R05 | Honest terminal states, bounded history-assisted queries, and partial/error display | +| F15 | Evaluation denominator/configuration/support limitations | R01, R08 | Answerable/unanswerable results are separate; shared configuration and evidence annotations are explicit | +| F16 | Legacy embedding API and serial/model-specific adapter gaps | R07 | Batch/query/document encoding, model identity, timeout and fallback contracts work | +| F17 | Stack compatibility and reproducibility gaps | R01, R07 | One reproducible baseline; actual client/server/provider compatibility is exercised | +| F18 | No real annotated meteorology corpus | R06, R08 | Owner-selected material and reviewed questions support professional acceptance | + +## 7. Implementation increments + +Order is R01 -> R02 -> R03 -> R04 -> R05 -> R06 -> R07 -> R08. R07's experiment track is conditional and can be explicitly deferred; its adapter/compatibility work is considered separately. Split an increment only when its recovery or verification boundary requires it, and record the resulting scope rather than proceeding to multiple increments implicitly. + +### R01 — Establish the runnable, recoverable baseline + +**Entry:** owner authorizes R01; refresh live Git and environment.\ +**Purpose:** make future changes measurable without destroying existing state. + +- Inventory settings, commands, dependency declarations, lockfile use, runtime paths, task lifecycle and API consumers. +- Resolve `.env` loading, configuration construction/validation and inert imports with explicit lifecycle management where required to establish the baseline. +- Define a loopback-only personal-use startup profile; external network exposure is a separate scope decision. +- Establish a side-effect-reviewed verification path that retains temporary artifacts and never invokes forbidden filesystem cleanup, including pytest/plugin defaults, OCR helpers and build tools. +- Reproduce the relevant investigation probes and save their inputs/outcomes as qualified baseline evidence. Do not label expected baseline defects as fixed. +- Define the production/evaluation configuration contract and metric denominators before comparisons; preserve legacy metric output or specify its compatibility transition. +- Determine which currently declared dependencies and services are actually needed for keyword-only operation. New packages require owner approval. +- Resolve OpenSpec tooling availability through an authorized setup choice; do not hand-author a falsely CLI-validated change. + +**Exit:** documented local startup and configuration; repeatable safe deterministic checks; known defects classified; runtime/data snapshot and recovery method identified. Real-service readiness may be BLOCKED with an explicit scope limit.\ +**Recovery:** restore configuration/startup behavior using focused corrective commits; retain all user data and prior receipts.\ +**Primary files:** configuration, composition, dependency/CI definitions, test setup and evaluator as justified by the baseline findings. + +### R02 — Make retrieval boundaries and reports consistent + +**Entry:** R01's applicable baseline and verification path are complete.\ +**Purpose:** one retrieval contract for all public entry points. + +- Introduce the minimal common scope and per-request report structures. +- Apply catalog eligibility and scope to search, chat and stream, including the keyword fallback. +- Replace shared request status with returned reports; align provider protocol signatures and consumers. +- Correct backend filtering and stable pagination without changing QA context selection silently. +- Preserve public response compatibility where possible; document intentional corrections and add negative boundary tests. + +**Exit:** F01-F04 behavior evidence passes; overlapping requests, empty scope, invalid sources and scoped fallback are covered.\ +**Recovery:** retain compatible endpoints and schemas; revert only this increment's behavior through corrective commits if needed.\ +**Primary files:** retrieval/protocols, API schemas/routes, QA consumers and corresponding tests. + +### R03 — Repair indexing, tasks and publication + +**Entry:** R02's eligibility contract exists.\ +**Purpose:** task success means an actually usable, coherent index state. + +- Persist an input receipt and task/source association before processing; associate retries with their parent and record their actual recoverability. +- Distinguish equivalent successful work from failed/stale work during duplicate detection. +- Recover queued and running tasks after restart, using one local ownership strategy; enforce the chosen single-process job model instead of accidentally supporting multiple competing runners. +- Implement staged publication and explicit lexical/vector capability readiness; define failure recovery across storage boundaries. +- Make logical removal/update invalidate every retrieval entry point consistently without deleting master files. +- Bound upload size, queued input, concurrent index work and retained future/report state based on resource behavior; serialize or isolate index construction as needed. +- Add failures before/after each publication boundary, retry checks and restart checks. Test on isolated retained artifacts. + +**Exit:** F05-F08 behavior evidence passes; no unexecuted recovery is reported successful; previous usable evidence survives failed publication.\ +**Recovery:** retain previous active generation and input receipts; restore its catalog reference through the documented procedure.\ +**Primary files:** ingestion, catalog, task service, index composition, API task behavior and tests. + +### R04 — Unify revisions, evidence locations and migrations + +**Entry:** R03's publication and recovery contract exists.\ +**Purpose:** a source remains identifiable through edits, backend changes and citations. + +- Design a minimal stable-document/revision/evidence schema and a legacy-ID mapping. +- Preserve source provenance, page sequence versus printed page labels where available, heading path and evidence relationships. +- Hydrate retrieved identifiers from the catalog rather than reconstructing partial source records in each backend. +- Record the actual parser/chunker/encoding configuration and validate vector-space dimension/model compatibility before activation. +- Separate reparse, rechunk and re-embed operations; a reindex command must state which operation it performs. +- Write a repeatable migration with preconditions and postconditions. Verify it on a copy with references, interrupted migration, repeated invocation and restoration to the prior state. +- Actual migration of existing user data needs an explicit authorized input scope and reviewed recovery receipt; development fixtures do not authorize touching the live catalog. + +**Exit:** F09 behavior evidence passes; legacy sources resolve; old and new revisions cannot silently alias; migration/recovery works on retained copies.\ +**Recovery:** retained schema/data snapshot and source-ID mapping; no dropping files or destructive resets.\ +**Primary files:** catalog/domain models, ingestion/parsing, vector hydration, schemas and migration tests. + +### R05 — Refactor evidence reading, answers and the personal interface + +**Entry:** R02 scope and R04 source identity are stable.\ +**Purpose:** give the owner an inspectable evidence workflow. + +- Index meaningful titles/aliases while preserving original text and heading hierarchy. +- Add focused section/neighbor reading under a recorded context budget; retain definitions, formula conditions and source locations where present. +- Separate ordering from evidence sufficiency; create hard negatives with overlapping terms but absent answer evidence. Avoid promising perfect automatic refusal. +- Validate citation membership and report used citations separately from candidates. Scientific support still requires domain acceptance. +- Align streaming/non-streaming generation, partial/failure states, readiness display and safe source opening within permitted roots. Treat source text, filenames and model output as untrusted display content; verify path and browser-rendering boundaries. +- Use bounded conversation history to resolve follow-up references without treating historical answers as source evidence. +- Add source reading/opening, scope selection and failure feedback to the existing browser console. Introduce a frontend framework only if concrete interface needs justify it and dependency approval exists. + +**Exit:** F11-F14 checks pass; the owner can inspect a citation's source revision; deterministic checks do not masquerade as scientific correctness.\ +**Recovery:** compatible existing routes and a retained UI behavior baseline; fall back to source search/read if generation is unavailable.\ +**Primary files:** parsing/chunking, evidence/QA/session services, API schemas/routes, web console and tests. + +### R06 — Integrate selected Atmos-RAG material one way + +**Entry:** R03-R05 source and evidence lifecycle works; owner identifies permitted source roots.\ +**Purpose:** turn the archive's conventions into implemented intake behavior. + +- Implement an explicit allowed-root source adapter with path/symlink escape checks and no master-file writes. +- Parse the needed frontmatter fields; define missing/invalid metadata behavior and report exclusions. +- Enforce directory exclusions, `rag_index: false`, domain separation, draft/review policy and evaluation/training-output isolation before indexing. +- Detect content changes, source disappearance and renames; retain original-revision provenance and publish updates through R03/R04. +- Keep methodology PDFs in a separate scope. Teaching notes and templates are not automatically meteorology evidence. +- Use synthetic fixtures first. Integrate only the owner's named real corpus; no hidden scan of the entire vault. +- Turn applicable question seeds into concrete questions with reviewed source locations. Retain unreviewed seeds separately and label unavailable material honestly. + +**Exit:** F10 checks pass; controlled import/update/unavailable-source scenarios work; source masters are unchanged. F18 domain readiness remains BLOCKED until real material and annotation exist.\ +**Recovery:** disable the adapter or restore the previous active catalog generation; originals are untouched.\ +**Primary files:** a focused source adapter, metadata intake, local config, catalog integration and tests. Atmos-RAG originals are read-only unless separately authorized. + +### R07 — Modernize adapters and evaluate optional components + +**Entry:** R01-R06 applicable contracts pass; verify current official compatibility information again.\ +**Purpose:** update necessary interfaces, then retain only justified complexity. + +**Contract track:** + +- Migrate Ollama embedding to the current batch-capable interface, with explicit query/document semantics for the configured model. +- Reuse clients where useful; validate batch output shape, finite values, dimensions, model identity, timeout and readiness behavior. +- Separate optional vector/OCR/model dependencies and document a reproducible lexical installation path; preserve declared contracts and ask before new packages. +- Align a chosen Milvus client/server/Lite profile and Docker model connectivity. A container's loopback address is not the host's model endpoint. +- Retain explicit degraded capabilities and run real adapter checks on authorized local services before declaring them qualified. + +**Conditional experiment track:** + +- Compare terminology expansion, multilingual embedding, reranking and generation models on the same annotated corpus and question split. +- Record source/model revisions, candidate and context budgets, quality, latency, memory and calls. Do not choose defaults from advertised leaderboards. +- Keep the existing backend until replacement has a concrete operational benefit. A new vector store or framework is a separate scope/dependency decision. +- If annotated domain evidence is absent, defer the experiments explicitly. Do not block contract repairs on a fake performance comparison. + +**Exit:** F16-F17 evidence passes for the selected supported profiles; optional defaults have evidence or remain unchanged/deferred. Required real-service checks that remain BLOCKED keep the corresponding qualification and increment exit pending; deterministic adapter tests do not close them.\ +**Recovery:** retained adapter/configuration path and prior index generation; model changes activate a distinct encoding space.\ +**Primary files:** provider/vector/rerank adapters, dependency/lock definitions, deployment and evaluation profiles. + +### R08 — Verify integrated personal use and close the refactoring program + +**Entry:** required prior increments are verified; conditional experiments have an explicit disposition.\ +**Purpose:** independently judge engineering readiness and actual personal usefulness. + +- Exercise startup, keyword-only mode, dependency failure, selected import, revision update, retry, restart, logical removal, search/chat/stream scope, source opening and partial/error display on the final content. +- Verify the final environment/installation instructions and recovery steps on retained copies. Record the actual OS/runtime; do not imply untested Windows/Linux support. +- Evaluate labeled real questions separately for retrieval, complete-evidence coverage, answer support, refusal and scientific conditions. +- Have the owner perform source-based manual acceptance on real learning/research tasks. Record acceptance, requested corrections and missing inputs without fabricating approval. +- Run the workflow consistency audit in section 11 and reconcile status, current changes, main specifications, documentation and verification receipts. +- Archive only actually completed OpenSpec changes using the applicable workflow. Archive means preserving history, not deleting files or branches. + +**Exit:** engineering acceptance and domain/manual acceptance are reported separately. If domain inputs are missing, report the engineering increment's actual completion and leave product/domain acceptance BLOCKED; do not mark the whole program complete.\ +**Recovery:** retain prior releases/configuration/data snapshots and use verified restoration or corrective commits.\ +**Primary files:** evaluation/tests, usage/recovery documentation, status and completed change records. + +## 8. Per-increment workflow + +1. **Resume and inspect:** read applicable instructions, status, the plan's relevant increment, its OpenSpec artifacts, then live Git/code and receipts. Reconfirm the actual active scope; do not infer it from the last conversation summary. +2. **Scope the authorized increment:** record the user's reference, objective, inputs, affected responsibilities, exclusions, compatibility changes, verification, recovery and completion conditions. Routine implementation choices inside this scope do not require repeated permission. +3. **Prepare the change:** use the existing OpenSpec workflow where applicable. Check current CLI/schema behavior and read its generated instructions. No active change is created by this planning package. A top-level roadmap does not replace implementation proposal/design/delta specs/tasks. +4. **Reproduce and establish baseline:** run meaningful, side-effect-reviewed checks. Record expected defects and blocked real dependencies before editing. +5. **Implement a coherent correction:** follow contracts and preserve unrelated work. Update affected specs/tests/documentation when behavior intentionally changes. Avoid incidental model swaps or broad file rearrangement. +6. **Verify final content:** run checks appropriate to consequences, including relevant failure/recovery cases. Repeat affected verification after later edits. Do not rerun unrelated expensive checks without a reason. +7. **Review and record:** check scope, contracts, sources, authority, recovery and evidence. Update the current change's tasks and the status record; store the verification receipt. Missing checks keep their own BLOCKED state. +8. **Commit and deliver:** inspect the staged diff; commit only the completed task's changes with the repository's Conventional Commit practice. Report the final hash, result, limits and next eligible action. No extra commit is needed solely to record its own hash. +9. **Remote and next increment:** push only when explicitly authorized for the current destination. Do not infer merge, deployment, release or the next increment from push permission. Because work proceeds one increment at a time, wait for the owner's next instruction after delivery. + +If a meaningful blocker is independent of some authorized work, finish the safe independent part and retain the blocker. Do not claim a complete increment when a known material acceptance condition is unresolved. A plan revision cannot retroactively widen the user's authorization. + +## 9. Boundaries and recovery + +| Action | Default handling | +| --- | --- | +| Read project code and named source metadata | Within the relevant authorized task; respect secrets/private content | +| Edit code/tests/docs in an authorized increment | Proceed within scope; preserve unrelated work | +| Add a dependency | Ask first, with purpose and alternatives | +| Install existing declared dependencies | Review side effects; proceed within authorized setup scope | +| Delete files, folders, temporary data, build artifacts or entire files through patches | Do not perform; explain the need and leave the action to the owner | +| Change an existing live user database/index | Prepare a reviewable migration/recovery result first; require authorization for named live inputs | +| Read a wider private source root or upload its content | Require scope authorization; do not infer from an adapter design | +| Call cloud models or external services with private input | Require explicit destination/input authorization | +| Download new weights or run resource-heavy experiments | Define model/resource scope first; ask when costs or impact are materially uncertain | +| Commit complete task changes | Commit only task paths after review and checks | +| Push, merge, deploy or release | Separate explicit authorization as applicable | +| Delegate to agents or message another chat | Only when the user or applicable instructions explicitly authorize it | +| Update global Codex memory | Only after a direct request to update it; project handoff notes are different | + +Recovery avoids destructive Git reset, force-push and history rewriting. Correct published mistakes with new commits. Retain previous index generations and backups; test restoration without deleting user originals. Under the current filesystem rule, test cleanup is also a prohibited operation, not an exception because files are temporary. + +## 10. Project memory and long-running work + +Use three different categories: + +| Category | Record | Rule | +| --- | --- | --- | +| Engineering memory | Status, current change design/tasks, verification receipts and Git | Records scope, decisions, progress, failures and recovery; never grants permission | +| Scientific knowledge | Original sources, revisions, reviewed notes and evidence locators | Preserve provenance; distinguish original text, personal inference and generated text | +| Conversation history | Bounded application session data | Helps resolve a question; never substitutes for source evidence or authorization | + +For project recovery, [REFACTOR_STATUS.md](docs/REFACTOR_STATUS.md) is the single current-state entry. It should remain short: current increment, user authorization reference, branch/baseline, decision/task links, latest applicable verification, unresolved issues and the next action. Store detailed history in the change and Git rather than continually growing duplicate state files. + +Before ending an implementation session, persist: + +- What actually changed and what remains unfinished. +- The precise resume point and any interrupted operation. +- Final content/commit identity for each applicable verification receipt. +- Actual commands/environment, outcomes and unverified behavior. +- Known limitations and the reason for a blocked condition. +- Any new user constraint, with its scope and direct reference. + +On resume: + +- Read instructions -> status -> relevant plan increment -> current OpenSpec artifacts -> live Git/code -> verification receipts. +- Treat old hashes, branch state, service availability and source paths as snapshots; refresh relevant facts. +- If authorization is unclear, continue independent read-only preparation and request only the missing decision. +- If a checked task lacks required evidence or the code changed afterward, reopen the affected verification. +- If scope changed, reconcile proposal/design/delta requirements/tasks and status before dependent implementation. + +Keep project records in Git free of source excerpts, credentials, private absolute paths and raw session logs. Detailed runtime fixtures belong in a reviewed local artifact root excluded from source commits. Verify exclusion before creating private records; do not assume a directory is ignored. + +## 11. Verification and workflow consistency audit + +### 11.1 Outcome vocabulary + +| Outcome | Meaning | +| --- | --- | +| PASS | A named check ran on identified content and met its condition | +| FAIL | A named check ran and did not meet its condition | +| BLOCKED | A required check/action could not run; state the cause and consequence | +| NOT_APPLICABLE | The check/action is outside this increment, with a reason | + +Use PLANNED, ACTIVE, VERIFYING, COMPLETE and DEFERRED for work disposition, not as verification results. COMPLETED DOCUMENTATION is not COMPLETED IMPLEMENTATION. A blocked live adapter check cannot be converted into PASS by a fake test. An expected pre-fix failure is baseline evidence, not a regression fix. + +### 11.2 Verification receipt + +Each implementation receipt should identify increment/change, tested commit or content hashes, baseline, environment, allowed inputs, commands, results, retained artifacts, skipped/blocked checks, recovery evidence and manual acceptance. Logs alone do not prove which content was tested. + +Inspect tests, plugins, subprocesses and tools for side effects before execution. Prevent or avoid forbidden cleanup; retain fixtures. Do not run the normal full suite first and inspect its effects afterward. R01 selects the usable strategy instead of importing a runner or adding a dependency merely because another project used it. + +A documentation-only change needs link/structure/scope review, not unrelated application tests. Formal OpenSpec validation is a separate check and must be labeled BLOCKED when its CLI is unavailable. + +### 11.3 Audit dimensions + +| Audit | Question | Failure handling | +| --- | --- | --- | +| Scope | Does every change belong to the authorized increment? | Remove the change from this increment's scope without deleting unrelated work; re-scope separately | +| Authority | Are user instructions distinguishable from generated plans and memory? | Stop the affected action; locate or request missing authorization | +| Dependency | Are entry conditions satisfied by evidence? | Keep the dependent action pending; finish independent work | +| Contract | Do proposal, design, delta specs, implementation, API, UI and tests agree? | Reconcile intended behavior and affected checks | +| State | Do tasks/status match actual files and Git? | Correct the record; do not fabricate completion | +| Evidence | Does each PASS identify what ran on final content? | Re-run affected verification or mark it unavailable | +| Data | Are source revision, domain, review state and evidence/test isolation preserved? | Reject contaminated or ambiguous intake/publication | +| Recovery | Can failure/interruption preserve and restore a usable state? | Keep migration/publication readiness incomplete | +| Privacy | Are logs, fixtures and published docs within the permitted input/destination? | Stop exposure and correct the affected artifact safely | +| Delivery | Are commit, push, merge and release accurately separated? | Report actual delivery status and do not advance | +| Memory | Can another session resume without hidden chat-only state? | Update the authoritative handoff links and unresolved items | +| Proportionality | Does a mechanism solve a demonstrated problem at personal-use scale? | Simplify the proposal before implementing it | + +### 11.4 Scenarios the workflow must handle + +Audit the normal completion path and these disruptions: + +- A session ends with partially edited files and no commit. +- A user narrows scope while implementation is running. +- Two records disagree about the current increment or acceptance. +- A dependency or service is unavailable after deterministic checks pass. +- A test or plugin would delete temporary files. +- A real corpus or professional annotation has not been provided. +- Index publication or schema migration stops between storage steps. +- A completed check's content is later edited. +- New files contain private source paths or material. +- A push is not authorized or is rejected because of divergence. +- An earlier increment is reopened by a later regression. +- An optional experiment is deferred without pretending that required behavior passed. + +Planning review tests whether the rules cover these scenarios. Implementation review must also exercise the relevant actual behavior. A same-agent review is not described as an independent review. + +## 12. Program completion and the first next step + +The program is complete only when the required refactoring increments satisfy their exits, source integration respects its boundaries, the final personal-use scenarios pass, and the owner accepts the real domain workflow. Optional research experiments can remain explicitly deferred. Engineering readiness and professional usefulness are separate conclusions. + +The first implementation step is R01 only. Its initial work packet is: refresh Git/runtime/tooling, map startup/configuration and evaluation contracts, design the retained-artifact verification path, and reproduce the relevant baseline failures. It does not authorize migrating the vault, swapping models, pushing or starting R02. + +This plan's current delivery state and the next authorized action are maintained in [REFACTOR_STATUS.md](docs/REFACTOR_STATUS.md). diff --git a/docs/REFACTOR_STATUS.md b/docs/REFACTOR_STATUS.md new file mode 100644 index 0000000..5502f25 --- /dev/null +++ b/docs/REFACTOR_STATUS.md @@ -0,0 +1,72 @@ +# Refactoring Status and Resume Point + +Updated: 2026-10-02, America/New_York + +This is the current-state entry for [the refactoring plan](../REFACTOR_INTEGRATION_PLAN.md). User instructions grant authority; this file records that authority and never creates it. + +## Current work packet + +| Field | Current value | +| --- | --- | +| Packet | PLAN-00: write and review the refactoring/integration plan | +| Authorized scope | Markdown planning and project recovery records in Insight; workflow consistency review | +| User reference | 2026-10-02 request to proceed one item at a time, using software-engineering and personal-use standards, with refactoring, boundaries, implementation planning, memory and consistency review | +| Planning disposition | COMPLETE | +| Implementation disposition | NOT_STARTED | +| Active implementation increment | NONE | +| Branch | `codex/refactor-integration-plan` | +| Application baseline | `abb59d7515119e9dcc4ee13bc307cbdc4bed8841` | +| Delivery scope | Three new Markdown files; existing application/specification/source contents remain unchanged | +| Remote delivery | Not authorized; retain the planning commit locally | +| Next eligible increment | R01, after the owner instructs work to start | + +## Authoritative references + +- [Plan, contracts and increment sequence](../REFACTOR_INTEGRATION_PLAN.md). +- [Planning consistency review and verification limits](reviews/2026-10-02-refactor-plan-consistency.md). +- [Current main behavior specifications](../openspec/specs). +- Existing user working instructions and any applicable future `AGENTS.md` take precedence over this record. + +No implementation OpenSpec change exists for this program yet. The roadmap is not a completed proposal/design/delta-specification/task package. A future implementation packet must use the applicable workflow and its live CLI/schema instructions. + +## Implementation ledger + +| Increment | Disposition | Completion evidence | +| --- | --- | --- | +| R01 | PLANNED | None; baseline and safe verification setup are first | +| R02 | PLANNED | None | +| R03 | PLANNED | None | +| R04 | PLANNED | None | +| R05 | PLANNED | None | +| R06 | PLANNED | None; real source-root/corpus scope must be named | +| R07 | PLANNED | None; adapter contracts and optional experiments are separate tracks | +| R08 | PLANNED | None; engineering and professional acceptance are separate | + +## Known constraints and verification gaps + +- OpenSpec CLI is unavailable in the current shell; formal validation is BLOCKED. No CLI was installed in PLAN-00. +- The ordinary local Python environment lacks required application/test dependencies. PLAN-00 is a documentation-only delivery and does not claim application tests ran on it. +- Prior syntax/lint, pure-memory tests and GitHub CI apply to the investigation/application baseline. They do not qualify future refactoring code, adapters or professional correctness. +- Existing 144 file-mode differences are unrelated work. They must not enter this packet's staged diff or later refactoring commits accidentally. +- Real Ollama/Milvus, complex PDF parsing and cancellation propagation require later real checks. +- There is no established real meteorology corpus or annotated domain test set. The twelve Atmos-RAG seeds are not formal evaluation data. +- Filesystem deletion is prohibited, including temporary/test cleanup. A usable retained-artifact strategy must be established before affected tests run. +- Original Atmos-RAG/Obsidian material is preserved. No wider scan, source rewrite, migration, model download, cloud call or global Codex-memory update is authorized by PLAN-00. + +## Resume instructions + +1. Read applicable user/project instructions, this file, the relevant plan increment, and any current OpenSpec artifacts. +2. Inspect live Git, configuration/code and referenced evidence. Treat dates, branches, hashes and availability in this file as snapshots. +3. If the owner authorizes R01, prepare only its work packet. Follow the plan's entry/exit and side-effect-review rules; use routine judgment within that scope. +4. If no implementation instruction has arrived, keep implementation NOT_STARTED. Do not infer permission from this plan, its completion, or a future push authorization. +5. Before each session ends, update the active scope, unfinished work, verification identity/limits and precise next action; keep details in the current change and receipts. + +## Locating the planning delivery commit + +The delivery commit is identified in Git history rather than copied into a self-referential tracking commit: + +```sh +git log -1 --format='%H %s' -- REFACTOR_INTEGRATION_PLAN.md docs/REFACTOR_STATUS.md docs/reviews/2026-10-02-refactor-plan-consistency.md +``` + +At later increments, record existing relevant commit references and receipt links. Do not create an additional commit solely to record a document's own new hash. diff --git a/docs/reviews/2026-10-02-refactor-plan-consistency.md b/docs/reviews/2026-10-02-refactor-plan-consistency.md new file mode 100644 index 0000000..36042d5 --- /dev/null +++ b/docs/reviews/2026-10-02-refactor-plan-consistency.md @@ -0,0 +1,75 @@ +# Refactoring Plan Consistency Review + +Review date: 2026-10-02, America/New_York\ +Packet: PLAN-00\ +Reviewer: authoring agent; same-agent review, not independent acceptance\ +Scope: planning artifacts only + +Reviewed artifacts: + +- [Refactoring and integration plan](../../REFACTOR_INTEGRATION_PLAN.md). +- [Current status and resume point](../REFACTOR_STATUS.md). +- This review record. + +## Review basis + +The owner requested a Markdown plan in Insight, focused on refactoring for personal use, including workflow, boundaries, implementation planning, memory management and consistency review. The planning package follows the existing working instructions and leaves implementation pending. + +The current application baseline is `abb59d7515119e9dcc4ee13bc307cbdc4bed8841`. Existing main specifications were read as current contracts. They were not changed to pretend the proposed behavior already exists. No application change, source migration, installation, model call, remote delivery or global memory update is part of this packet. + +## Consistency findings + +| ID | Dimension | Result | Evidence or resolution | +| --- | --- | --- | --- | +| C01 | Product scope | PASS | One local user and one application home; optional enterprise/research mechanisms excluded from the baseline | +| C02 | Refactoring focus | PASS | Findings map to R01-R08 behavior and recovery work; technology swaps are conditional | +| C03 | Increment dependencies | PASS | Ordered entry/exit criteria; only R01 is next; required blocked R07 qualification keeps its exit pending; optional experiments have a separate disposition | +| C04 | Authorization | PASS | Planning is authorized, implementation remains NOT_STARTED; documents do not grant permissions | +| C05 | Filesystem preservation | PASS | No deletion exception for fixtures/OCR/build cleanup; source masters stay preserved | +| C06 | Source and privacy boundaries | PASS | Allowed-root one-way intake, evaluation exclusions and no private absolute paths in this package | +| C07 | State and memory ownership | PASS | Status owns live progress; plan owns direction; OpenSpec owns change contracts; receipts own results | +| C08 | Evidence semantics | PASS | PASS/FAIL/BLOCKED/NOT_APPLICABLE defined; prior checks, mock behavior and real/professional acceptance distinguished | +| C09 | Data recovery | PASS | Index publication, revision lineage and copy-based recovery defined; retained active revisions are distinguished from failed staged replacements; no destructive Git rollback | +| C10 | OpenSpec consistency | PASS | Existing main contracts remain authoritative; intended deltas require future change artifacts; no fabricated CLI result | +| C11 | Delivery separation | PASS | Local commit permitted; push/merge/deployment/release remain separate; unrelated modes excluded | +| C12 | Personal-use proportionality | PASS | Minimal records and modular existing app; no automatic framework, vector-store or multi-agent expansion | + +These are manual planning-coherence findings. They do not prove implementation behavior or professional answer quality. + +## Workflow disruption review + +| Scenario | Expected process behavior | Review result | +| --- | --- | --- | +| Partial edits at interruption | Status records unfinished work and live Git is inspected before resume | PASS: covered by workflow and project-memory rules | +| User narrows scope | Reconcile affected change/task records; do not treat the plan as authority | PASS: covered | +| Plan/status/specification disagreement | Record and reconcile the discrepancy before completion | PASS: covered | +| Services unavailable | Finish independent authorized work; retain named BLOCKED real checks | PASS: covered | +| Test cleanup would delete files | Avoid the path until a compliant strategy exists; no temporary-file exception | PASS: covered | +| Real corpus/annotation missing | Use qualified fixtures for engineering work; keep professional acceptance BLOCKED | PASS: covered | +| Interrupted publication/migration | Restore retained usable generation/copy using verified recovery procedure | PASS: procedure required in R03/R04, runtime execution pending | +| Content edited after verification | Reopen affected checks and identify final content | PASS: covered | +| Private content in a new record | Stop exposure and correct the affected record; verify local artifact exclusion | PASS: covered | +| Push absent or rejected | Keep local delivery or inspect divergence; no force-push or inferred integration | PASS: covered | +| Regression reopens a completed increment | Reopen applicable verification and reconcile status/tasks | PASS: covered | +| Optional experiment deferred | Keep its disposition explicit and separate from required contract checks | PASS: covered | + +## Planning verification receipt + +| Check | Result | Scope and limitation | +| --- | --- | --- | +| Markdown links, anchors, fences, required records and increment/findings references | PASS | Read-only standard-library structural checks; not semantic or runtime proof | +| Manual plan/status/review consistency | PASS | Reviewed authorization, dependencies, record ownership and disruption coverage above | +| Existing file-content preservation | PASS | All baseline tracked contents remain unchanged; pre-existing mode changes preserved | +| Staged delivery scope | PASS | Exactly the three new Markdown artifacts; no application, dependency, specification or unrelated mode changes staged | +| Formal OpenSpec validation | BLOCKED | CLI unavailable; no OpenSpec implementation change created by this packet | +| Application tests and real adapters | NOT_APPLICABLE | Documentation-only packet; baseline investigation results are not reported as new implementation checks | +| Professional/manual product acceptance | NOT_APPLICABLE | Implementation has not started; real corpus and annotations are still absent | +| Remote push/merge/release | NOT_APPLICABLE | Not authorized in PLAN-00 | + +The author checks the actual final artifact contents and staged diff before committing. The local delivery commit binds the reviewed files; find its identity through the command in the status record. No later content edits are covered by this receipt without renewed verification. + +Verification method: an inline `python3 -B` standard-library checker reads the three artifacts, checks UTF-8/newlines, balanced fences, local link/anchor targets, ordered R01-R08 sections, F01-F18 ownership, status/review identifiers and absence of private absolute paths. It compares every pre-existing tracked file's bytes with the baseline and checks that only the pre-existing 144 mode differences remain. `git diff --cached --check` and a staged-path comparison verify the exact delivery scope. These checks write no fixtures and invoke no cleanup. + +## Disposition + +The planning package is complete. Application refactoring remains NOT_STARTED. R01 is the next eligible implementation increment after the owner's instruction. No whole-program completion, runtime qualification or scientific acceptance is implied. From 43b4136f52551fd272d91dea7ef8830b31af2668 Mon Sep 17 00:00:00 2001 From: zhenkun Date: Sat, 17 Oct 2026 03:00:05 -0400 Subject: [PATCH 2/5] docs: rewrite README and add English version --- README-en.md | 218 ++++++++++++++++++++++++++++++++ README.md | 351 +++++++++++++++++++++------------------------------ 2 files changed, 359 insertions(+), 210 deletions(-) create mode 100644 README-en.md diff --git a/README-en.md b/README-en.md new file mode 100644 index 0000000..32788a0 --- /dev/null +++ b/README-en.md @@ -0,0 +1,218 @@ +# Insight + +**Local document search and question answering for meteorological reference materials.** + +[简体中文](README.md) · [Report an issue](https://github.com/zhenkun26/Insight/issues) + +[![CI](https://github.com/zhenkun26/Insight/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/zhenkun26/Insight/actions/workflows/ci.yml) +[![Python](https://img.shields.io/badge/Python-3.11%2B-blue)](pyproject.toml) +[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE) + +Insight turns PDF, Markdown, and TXT files into a searchable knowledge base. Find relevant passages, then use a local model to generate answers with source information. A web console and HTTP API support organizing meteorological documents, consulting procedures, and experimenting with RAG retrieval configurations. + +Start with BM25 keyword search, which needs no model service, then add Ollama and Milvus for answer generation, vector search, and reranking. Documents, indexes, and sessions are stored locally by default. Running models through a local Ollama service requires no cloud model API key. + +## Features + +- **Document ingestion**: Import `.pdf`, `.md`, `.markdown`, and `.txt` files with available PDF page numbers and Markdown headings. Optional OCR handles PDF pages where text extraction returns no text. +- **Hybrid retrieval**: Combine BM25 and optional vector results with reciprocal rank fusion (RRF), followed by optional keyword-based or Ollama-based reranking. +- **Answers with sources**: Return retrieved passages, filenames, chunk IDs, and available page or section information. Return an explicit `refused` status when no passage passes the threshold. +- **Web console**: Upload documents, follow indexing progress, search passages, receive streamed answers, and inspect sources on one page. +- **Indexing and document management**: APIs expose job status, retries for eligible failed jobs, cancellation of queued jobs, and source/tag metadata editing. +- **Visible runtime behavior**: Inspect request trace IDs, stage timings, and retrieval fallback states. Vector-search or reranking failures preserve available baseline retrieval results. +- **Retrieval evaluation**: Compare BM25, vector, and hybrid retrieval with different reranking configurations using the included script. + +## Quick start + +The commands below use a macOS / Linux shell. You need **Python 3.11+**; repository CI uses Python 3.11. Installing dependencies requires network access. Keyword search does not need Ollama or a separate Milvus service at runtime. + +### 1. Install and start + +```bash +git clone https://github.com/zhenkun26/Insight.git +cd Insight + +python3 -m venv .venv +source .venv/bin/activate +python -m pip install -e ".[dev]" + +MILVUS_URI= python -m uvicorn app.main:app --host 127.0.0.1 --port 8000 +``` + +Open the [web console](http://127.0.0.1:8000). The same service provides [interactive API documentation](http://127.0.0.1:8000/docs) and a [health endpoint](http://127.0.0.1:8000/health). + +This command explicitly disables the vector store and starts with BM25. Document ingestion and keyword search work without Ollama. If a chat request retrieves passages while Ollama is unavailable, it reports a model error. + +### 2. Import a document and search + +1. Upload [`data/sample_docs/typhoon-warning.md`](data/sample_docs/typhoon-warning.md) in the console. +2. Wait for the indexing job to show `succeeded` and the document to show `indexed`. +3. Enter **台风预警信号分为几级？** in the search box (“How many levels of typhoon warning signals are there?”). +4. Inspect the matching passages, sections, and filenames. + +You can also call the search API from another terminal: + +```bash +curl -sS http://127.0.0.1:8000/search \ + -H 'Content-Type: application/json' \ + -d '{"query":"台风预警信号分为几级？","top_k":5}' +``` + +The response includes `retrieval_results`, `retrieval_status`, and `stages`. The bundled documents are synthetic examples for trying the workflow, not official meteorological guidance. + +### 3. Enable local model answers + +Install and start [Ollama](https://docs.ollama.com/quickstart), then download the project's default generation model: + +```bash +ollama pull llama3.2:3b +``` + +The default Ollama address is `http://localhost:11434`. If the service is not running, start `ollama serve` in another terminal. Keep Insight running, ask a question in the console, or call: + +```bash +curl -sS http://127.0.0.1:8000/chat \ + -H 'Content-Type: application/json' \ + -d '{"query":"台风预警信号分为几级？"}' +``` + +For streaming, use `/chat/stream` and add `-N` to curl. The current console and answer prompts primarily use Chinese. The default model is a configuration example; evaluate answer quality against your own documents. + +## Optional configuration + +The application reads environment variables and **does not load `.env` automatically**. See [`.env.example`](.env.example) for configuration examples and [`app/core/config.py`](app/core/config.py) for actual defaults. + +To use a `.env` file, copy the example without overwriting existing settings, edit the values you need, and load it explicitly: + +```bash +cp -n .env.example .env +python -m uvicorn app.main:app --env-file .env --host 127.0.0.1 --port 8000 +``` + +Stop the previous Insight process before running this command. The example file sets `MILVUS_URI=data/milvus.db`, which attempts to enable Milvus Lite. Leave this value empty for keyword-only retrieval. + +| Setting | Code default | Purpose | +| --- | --- | --- | +| `LLM_BASE_URL` | `http://localhost:11434` | Ollama service address | +| `LLM_MODEL` | `llama3.2:3b` | Answer generation model | +| `MILVUS_URI` | Empty | Empty disables vector storage; a file path selects Milvus Lite, a service URI selects Milvus Server | +| `EMBEDDING_MODEL` | `nomic-embed-text` | Embedding model | +| `TOP_K` / `CANDIDATE_K` | `5` / `20` | Final result count / candidates per retrieval branch | +| `SCORE_THRESHOLD` | `0.01` | Score threshold after fusion or reranking | +| `VECTOR_SCORE_THRESHOLD` | `0.7` | Similarity threshold for vector candidates | +| `ENABLE_RERANK` / `RERANKER_MODEL` | `false` / Empty | Enable reranking; an empty model name selects keyword-based scoring | +| `OCR_ENABLED` / `OCR_LANGUAGE` | `false` / `eng` | PDF OCR switch and Tesseract language | +| `DATABASE_PATH` | `data/insight.db` | SQLite document, job, and session records | + +### Vector search and reranking + +Before enabling vector search, download the default embedding model: + +```bash +ollama pull nomic-embed-text +``` + +Set `MILVUS_URI=data/milvus.db` in `.env` and restart with the `--env-file` command above. Check the [Milvus Lite documentation](https://milvus.io/docs/milvus_lite.md) for platform requirements. + +If you previously imported documents, submit a reindex job and use its returned `job_id` to check progress: + +```bash +curl -sS -X POST http://127.0.0.1:8000/documents/reindex +``` + +After the job succeeds, inspect `retrieval_status.vector` in a search response: `ok` means vector retrieval completed normally for that request, `disabled` means it was not enabled, and `fallback:...` indicates a fallback. + +Reranking is disabled by default. Set `ENABLE_RERANK=true` for keyword-based scoring. Also set `RERANKER_MODEL` to a downloaded Ollama generation model to use model-based scoring. This makes one model call per candidate and adds query latency. + +### Scanned PDFs and containers + +- **OCR**: Install Poppler (`pdftoppm`), Tesseract, and the required language data, then set `OCR_ENABLED=true`. For scanned Chinese documents, select installed Chinese language data, such as `OCR_LANGUAGE=chi_sim+eng`. OCR only supplements pages with no extracted text. +- **Docker**: The repository includes a [Dockerfile](Dockerfile) and [Compose configuration](docker-compose.yml) for the API, Milvus, etcd, and MinIO. Run Ollama separately and set `LLM_BASE_URL` to an address reachable from the API container; container-local `localhost` does not refer to the host machine. Compose publishes host ports by default, so adjust the bindings for your intended access scope. + +## How it works + +```mermaid +flowchart LR + D["PDF / Markdown / TXT"] --> P["Parse and chunk"] + P --> B["BM25 index"] + P -. Optional .-> V["Ollama embeddings + Milvus"] + Q["Question"] --> R["Retrieval and RRF fusion"] + B --> R + V -.-> R + R --> K["Optional reranking and threshold filtering"] + K --> S["Search results"] + K --> G{"Chat: usable passages?"} + G -->|Yes| A["Ollama generation + source information"] + G -->|No| F["refused"] +``` + +FastAPI serves the backend. SQLite stores documents, chunks, jobs, and sessions; the BM25 index defaults to `data/bm25.json`, and uploaded originals are saved under `data/uploads/`. The backend serves the HTML, CSS, and JavaScript console directly, with no separate frontend build step. + +## API overview + +| Endpoint | Purpose | +| --- | --- | +| `GET /health` | Service and dependency status | +| `POST /documents/upload` | Upload a document using multipart field `file`; returns an indexing job | +| `GET /documents` | List documents with source, tag, and status filters | +| `PATCH /documents/{document_id}/metadata` | Edit source, tags, and description | +| `POST /documents/reindex` | Submit a reindex job | +| `GET /jobs/{job_id}` | Inspect job progress and errors | +| `POST /jobs/{job_id}/retry` | Retry an eligible failed job | +| `POST /jobs/{job_id}/cancel` | Cancel a queued job that has not started | +| `POST /search` | Search passages with `source`, `tag`, and `document_ids` scope filters | +| `POST /chat` | Return an answer, sources, and workflow status | +| `POST /chat/stream` | Stream stage, source, and text events over SSE | + +See the running service's [API documentation](http://127.0.0.1:8000/docs) for full request and response schemas. Chat API requests can include a `session_id` to retain bounded history. History provides conversational context, not document evidence. + +## Development and verification + +After installing the development dependencies above, run from the repository root: + +```bash +ruff check app tests scripts +pytest +python scripts/evaluate.py --retrieval-mode bm25 --output evaluation-bm25.json +``` + +The BM25 evaluation makes no model-service calls. Run `python scripts/evaluate.py --help` for other configurations. For example, compare hybrid retrieval using a downloaded embedding model: + +```bash +python scripts/evaluate.py \ + --retrieval-mode hybrid \ + --vector-backend memory \ + --embedding-model nomic-embed-text \ + --output evaluation-hybrid.json +``` + +Reports include hit rate, MRR, stage timings, and refusal-related metrics. Here, `refusal_accuracy` measures whether questions labeled for refusal produce empty retrieval results; it **does not measure the factual correctness of generated answers**. The bundled data is intended for regression checks and demonstrations. It does not replace human evaluation on real operational documents. + +Main code locations: + +| Path | Contents | +| --- | --- | +| [`app/api/`](app/api/) | HTTP routes and request orchestration | +| [`app/ingestion/`](app/ingestion/) | Parsing, chunking, and OCR | +| [`app/retrieval/`](app/retrieval/) | BM25, vector stores, and hybrid retrieval | +| [`app/services/`](app/services/) | Catalog, jobs, Ollama, QA, and sessions | +| [`app/web/`](app/web/) | Web console | +| [`scripts/evaluate.py`](scripts/evaluate.py) | Retrieval evaluation entry point | +| [`tests/`](tests/) | Automated tests | + +## Current limitations + +- **Sources support manual checking**: Prompts instruct the model to answer from documents and cite sources. The application does not yet validate each answer claim against its citations. Retrieval scores are not answer-confidence scores. +- **Search and chat scopes differ**: `/search` supports document scope filters; `/chat` and `/chat/stream` currently do not apply those filter parameters. +- **Designed for local use**: The API has no user authentication or permission isolation. The quick start binds to `127.0.0.1`; add access controls before sharing access. Configuring a remote model service sends the relevant queries or document passages to that service. +- **Format and domain boundaries**: Word documents are not supported. Review extraction quality for complex PDFs and scanned pages. Verify operational meteorological decisions against the applicable original documents. + +## Help and contributions + +Maintained by [zhenkun26](https://github.com/zhenkun26). Use [GitHub Issues](https://github.com/zhenkun26/Insight/issues) for questions, bugs, and feature requests. Pull requests for code and documentation improvements are welcome. + +For bug reports, include reproduction steps, your Python version, relevant configuration, and error status. Remove sensitive documents and credentials. Before submitting code changes, run lint, relevant tests, and any affected retrieval evaluations. + +## License + +Released under the [MIT License](LICENSE). Models and imported documents remain subject to their own licenses. diff --git a/README.md b/README.md index 3d0f6b0..21b7f9e 100644 --- a/README.md +++ b/README.md @@ -1,287 +1,218 @@ -# Insight +# 洞察者 · Insight -[![CI](https://github.com/zhenkun26/Insight/actions/workflows/ci.yml/badge.svg)](https://github.com/zhenkun26/Insight/actions/workflows/ci.yml) -[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE) +**面向气象业务文档的本地知识库检索与问答工具。** -Insight 是一个面向气象业务资料的 local-first RAG 应用。它支持在本地导入 PDF、Markdown 和 TXT 资料，使用 BM25 与向量检索进行混合召回，并返回带来源引用的问答结果。项目自带的气象资料为参考样例，用于验证检索与问答链路。 +[English](README-en.md) · [问题反馈](https://github.com/zhenkun26/Insight/issues) -Insight is a local-first RAG application for meteorological operational documents. It imports PDF, Markdown, and TXT files locally, performs hybrid retrieval with BM25 and vector search, and returns answers with source citations. The bundled meteorological documents are reference samples for verifying the retrieval and QA pipeline. +[![CI](https://github.com/zhenkun26/Insight/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/zhenkun26/Insight/actions/workflows/ci.yml) +[![Python](https://img.shields.io/badge/Python-3.11%2B-blue)](pyproject.toml) +[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE) -## 功能概览 +Insight 将 PDF、Markdown 和 TXT 资料转为可检索的知识库，帮助你查找相关段落，并通过本地模型生成附带来源信息的回答。它提供一个网页控制台和一组 HTTP API，适合整理气象业务资料、查阅流程说明，以及实验不同的 RAG 检索配置。 -- 在本地检索观测说明、预警信号说明和数据处理流程。 -- 关键词检索、语义检索、RRF 融合与可选重排序构成的混合召回链路。 -- 证据不足时明确拒答，避免模型自由补充事实。 +从无需模型服务的 BM25 关键词检索开始，再按需接入 Ollama 和 Milvus，启用回答生成、向量检索与重排序。资料、索引和会话默认保存在本地；使用本机 Ollama 模型时，无需云端模型 API 密钥。 -## 架构 +## 主要功能 -```mermaid -flowchart LR - A[PDF / Markdown / TXT] --> B[解析与分块] - B --> N[本地索引任务] - N --> C[SQLite 文档目录] - N --> D[BM25 索引] - N --> E[Ollama Embedding] - E --> F[Milvus / Milvus Lite] - Q[用户问题] --> G[BM25 + 向量召回] - F --> G - D --> G - G --> H[RRF 融合] - H --> I[可选 Rerank] - I --> R{相关性足够?} - R -->|否| K[明确拒答 + 阶段状态] - R -->|是| L[Ollama 生成] - L --> M[回答 + 来源引用 + trace] -``` +- **文档导入**：支持 `.pdf`、`.md`、`.markdown`、`.txt`，保留可提取的 PDF 页码与 Markdown 章节信息；可选 OCR 处理未提取到文本的 PDF 页面。 +- **混合检索**：BM25 关键词召回与可选向量召回通过 RRF（倒数排名融合）合并，可进一步使用关键词规则或 Ollama 模型重排序。 +- **带来源的问答**：返回检索片段、文件名、片段 ID 及可用的页码、章节；无片段通过阈值时返回明确的 `refused` 状态。 +- **网页控制台**：在同一页面上传资料、查看索引进度、搜索片段、接收流式回答并查看来源。 +- **索引任务与资料管理**：API 提供任务状态、符合条件的失败任务重试、排队任务取消，以及资料来源和标签编辑。 +- **运行状态可见**：返回请求追踪 ID、阶段耗时和检索降级状态；向量检索或重排序失败时保留可用的基础检索结果。 +- **可比较的检索评估**：使用内置脚本比较 BM25、向量和混合检索，以及不同重排序配置。 -## 核心功能 - -- 文档上传、列表、删除和重建索引。 -- PDF 页码、Markdown 标题和文本块元数据保留。 -- 可选扫描 PDF OCR：默认关闭，启用后只处理没有文本层的页面。 -- BM25 + 向量召回、向量分数归一化、RRF 融合、Top-K、双层阈值和可选 Rerank。 -- 可选 Ollama 模型重排：设置 `RERANKER_MODEL` 后按候选片段评分；模型失败自动保留 RRF 顺序。 -- `/chat`、`/chat/stream`、`/search`、文档管理和 `/health`。 -- Ollama、Milvus 和 Rerank 均通过 adapter 隔离，测试可使用 fake/mock。 -- 索引任务支持后台执行、状态轮询、失败重试、内容指纹幂等和模型版本变更提示。 -- 文档支持来源/标签过滤，搜索支持分页；问答支持可选会话上下文和 SSE 事件流。 -- RAG 工作流返回 query analysis、retrieval、rerank、relevance check、generation/fallback 阶段信息。 -- 评估脚本输出实际运行得到的 hit rate、MRR、拒答准确性和平均延迟；README 不预填性能指标。 -- 内置无 Node 依赖的本地 Web Console，可直接完成资料导入、混合检索和流式问答。 - -## 技术栈 - -Python 3.11+、FastAPI、Pydantic、Uvicorn、Ollama HTTP API、Milvus/Milvus Lite、BM25、SQLite、pytest、Docker Compose 和 GitHub Actions。 - -## 项目目录 - -```text -app/ -├── api/ # FastAPI routes -├── core/ # environment settings -├── ingestion/ # parsers and chunking -├── models/ # domain models -├── retrieval/ # BM25, vector, hybrid fusion -├── schemas/ # API schemas -├── services/ # catalog, Ollama, ingestion, jobs, sessions, QA -├── web/ # dependency-free local browser console -└── workflows/ # explicit RAG stage state -data/ -├── sample_docs/ # reference sample documents -└── uploads/ # local runtime uploads -scripts/evaluate.py -tests/ -Dockerfile -docker-compose.yml -pyproject.toml -``` +## 快速开始 -## 本地运行 +以下命令适用于 macOS / Linux shell。需要 **Python 3.11+**；仓库 CI 使用 Python 3.11。安装依赖需要网络，关键词检索运行时不需要 Ollama 或独立 Milvus 服务。 -需要 Python 3.11+。推荐使用虚拟环境和 uv： +### 1. 安装并启动 ```bash -uv venv +git clone https://github.com/zhenkun26/Insight.git +cd Insight + +python3 -m venv .venv source .venv/bin/activate -uv pip install -e ".[dev]" -cp .env.example .env -uvicorn app.main:app --reload +python -m pip install -e ".[dev]" + +MILVUS_URI= python -m uvicorn app.main:app --host 127.0.0.1 --port 8000 ``` -启动后打开 [http://localhost:8000/](http://localhost:8000/)，即可使用本地 Web Console。页面不需要 Node.js、前端构建工具或外部 CDN；它与 FastAPI 使用同源请求，上传后自动轮询索引任务，并在问答区展示 SSE 片段、来源和阶段状态。也可以使用 `/docs` 查看完整 API。 +打开 [网页控制台](http://127.0.0.1:8000)。[交互式 API 文档](http://127.0.0.1:8000/docs) 和 [健康检查](http://127.0.0.1:8000/health) 也由同一服务提供。 -不安装或不启动外部模型时，应用仍可启动并使用关键词检索；问答和向量召回会根据依赖状态返回明确结果。 +这条启动命令显式关闭向量存储，先使用 BM25。未启动 Ollama 时，文档导入和关键词检索仍可使用；有检索结果的问答请求会报告模型不可用。 -上传和重建索引默认返回后台任务，不阻塞 HTTP 请求。可通过 `/jobs/{job_id}` 轮询状态；进程重启后未完成的任务会被标记为可重试失败。 +### 2. 导入资料并检索 -## Ollama 模型准备 +1. 在控制台上传 [`data/sample_docs/typhoon-warning.md`](data/sample_docs/typhoon-warning.md)。 +2. 等待索引任务显示 `succeeded`，资料状态显示 `indexed`。 +3. 在检索框输入：**台风预警信号分为几级？** +4. 查看命中的片段、章节与文件名。 -安装 Ollama 后准备一个生成模型和一个 embedding 模型： +也可以在另一个终端调用检索接口： ```bash -ollama pull llama3.2:3b -ollama pull nomic-embed-text +curl -sS http://127.0.0.1:8000/search \ + -H 'Content-Type: application/json' \ + -d '{"query":"台风预警信号分为几级？","top_k":5}' ``` -通过 `.env` 配置 `LLM_BASE_URL`、`LLM_MODEL` 和 `EMBEDDING_MODEL`。模型名称、地址、Top-K、阈值和超时均通过配置注入，不硬编码在业务逻辑中。 +返回结果包含 `retrieval_results`、`retrieval_status` 和 `stages`。内置样例是合成演示资料，仅用于体验流程，不代表官方气象业务规范。 -### 可选扫描 PDF OCR +### 3. 启用本地模型问答 -扫描版 PDF 没有文本层时，可以额外安装本机 OCR 工具并显式开启： +安装并启动 [Ollama](https://docs.ollama.com/quickstart)，然后下载项目默认使用的生成模型： ```bash -# macOS -brew install poppler tesseract - -# Debian/Ubuntu -sudo apt-get install poppler-utils tesseract-ocr -``` - -中文资料还需要安装对应的 Tesseract 语言包，并确保 `pdftoppm`、`tesseract` 位于 `PATH`。在 `.env` 中设置： - -```dotenv -OCR_ENABLED=true -OCR_LANGUAGE=chi_sim+eng -OCR_TIMEOUT_SECONDS=30 -# 可选：指定 OCR 临时目录的父目录 -OCR_TEMP_DIR= +ollama pull llama3.2:3b ``` -OCR 默认关闭，不会影响普通 PDF、Markdown 和 TXT。启用后系统仅 OCR 文本层为空的 PDF 页面；工具缺失、超时或命令失败会让索引任务失败并返回 `ocr_unavailable`、`ocr_timeout` 或 `ocr_failed` 语义。启用 OCR 后建议对已有扫描资料执行一次重建索引。Docker 镜像不预装这些系统工具，需自行制作带 OCR 运行时的镜像。 - -## Milvus - -本地开发可以将 `MILVUS_URI` 指向 Milvus Lite 文件路径；使用 Docker Compose 时： +Ollama 默认服务地址为 `http://localhost:11434`。若服务尚未启动，可在另一个终端运行 `ollama serve`。保持 Insight 运行，回到控制台的问答区提问，或调用： ```bash -docker compose up -d milvus -uvicorn app.main:app --reload +curl -sS http://127.0.0.1:8000/chat \ + -H 'Content-Type: application/json' \ + -d '{"query":"台风预警信号分为几级？"}' ``` -完整 API 容器和 Milvus 依赖可一起启动： +需要流式输出时，改用 `/chat/stream` 并为 curl 添加 `-N`。当前控制台和问答提示词以中文为主。默认模型是配置示例，实际回答质量需要结合自己的资料验证。 -```bash -docker compose up --build -``` +## 可选配置 -首次启动或切换 embedding 模型后，建议调用重建索引接口。不同 embedding 模型的向量维度可能不同，不能直接复用旧集合。 +程序从环境变量读取配置，**不会自行加载 `.env`**。完整变量示例见 [`.env.example`](.env.example)，实际默认值见 [`app/core/config.py`](app/core/config.py)。 -## 导入文档 +如果使用 `.env`，请先复制示例文件（已有配置时保留原文件），再编辑所需值，并显式加载： ```bash -curl -X POST http://localhost:8000/documents/upload \ - -F "file=@data/sample_docs/typhoon-warning.md" +cp -n .env.example .env +python -m uvicorn app.main:app --env-file .env --host 127.0.0.1 --port 8000 +``` -curl http://localhost:8000/documents -curl -X POST http://localhost:8000/documents/reindex +运行前先停止上一条 Insight 启动命令。示例文件设置了 `MILVUS_URI=data/milvus.db`，会尝试启用 Milvus Lite；只使用关键词检索时，将该值留空。 -# 上传/重建响应中的 job_id -curl http://localhost:8000/jobs/ -curl -X POST http://localhost:8000/jobs//retry -curl -X POST http://localhost:8000/jobs//cancel +| 配置项 | 代码默认值 | 用途 | +| --- | --- | --- | +| `LLM_BASE_URL` | `http://localhost:11434` | Ollama 服务地址 | +| `LLM_MODEL` | `llama3.2:3b` | 回答生成模型 | +| `MILVUS_URI` | 空 | 留空关闭向量存储；文件路径用于 Milvus Lite，服务 URI 用于 Milvus Server | +| `EMBEDDING_MODEL` | `nomic-embed-text` | 向量化模型 | +| `TOP_K` / `CANDIDATE_K` | `5` / `20` | 最终返回数 / 每路召回候选数 | +| `SCORE_THRESHOLD` | `0.01` | 融合或重排序后的分数阈值 | +| `VECTOR_SCORE_THRESHOLD` | `0.7` | 向量候选的相似度阈值 | +| `ENABLE_RERANK` / `RERANKER_MODEL` | `false` / 空 | 开启重排序；模型名为空时使用关键词规则 | +| `OCR_ENABLED` / `OCR_LANGUAGE` | `false` / `eng` | PDF OCR 开关与 Tesseract 语言 | +| `DATABASE_PATH` | `data/insight.db` | SQLite 资料、任务与会话记录 | -# 更新来源和标签 -curl -X PATCH http://localhost:8000/documents//metadata \ - -H 'Content-Type: application/json' \ - -d '{"source":"reference","tags":["typhoon"],"description":"typhoon warning procedures"}' -``` +### 向量检索与重排序 -## 搜索与问答 +启用向量检索前，下载默认的向量化模型： ```bash -curl -X POST http://localhost:8000/search \ - -H 'Content-Type: application/json' \ - -d '{"query":"台风预警信号分为几级？","top_k":5}' +ollama pull nomic-embed-text +``` -curl -X POST http://localhost:8000/chat \ - -H 'Content-Type: application/json' \ - -d '{"query":"台风预警信号分为几级？"}' +在 `.env` 中设置 `MILVUS_URI=data/milvus.db`，按上面的 `--env-file` 命令重启。Milvus Lite 的平台要求见 [官方文档](https://milvus.io/docs/milvus_lite.md)。 -# 按标签过滤并分页 -curl -X POST http://localhost:8000/search \ - -H 'Content-Type: application/json' \ - -d '{"query":"预警","tag":"typhoon","offset":0,"top_k":5}' +如果此前已经导入资料，再提交一次重建索引任务，并通过返回的 `job_id` 查询进度： -# SSE 流式问答 -curl -N -X POST http://localhost:8000/chat/stream \ - -H 'Content-Type: application/json' \ - -d '{"query":"台风预警信号分为几级？","session_id":"demo-session"}' +```bash +curl -sS -X POST http://127.0.0.1:8000/documents/reindex ``` -问答响应包含 `answer`、`sources`、`retrieval_results`、`query`、`latency_ms` 和 `status`。来源包括文件名、页码（可用时）、章节和文本块 ID。没有达到阈值的上下文时，系统返回"当前知识库中没有足够信息"语义的拒答。 +确认任务成功后，在检索响应中检查 `retrieval_status.vector`：`ok` 表示该次向量检索正常，`disabled` 表示未启用，`fallback:...` 表示发生降级。 -完整响应还包含 `trace_id`、`stages` 和 `retrieval_status`。`/chat/stream` 使用 `text/event-stream`，事件包括 `start`、`retrieval`、`source`、`token` 和 `complete`；配置了 Ollama 时会使用原生 NDJSON 流式片段，并在连接异常时标记 fallback。排队中的索引任务可以取消，运行中任务不会被强制终止。会话历史仅辅助当前问题理解，不替代当前轮次的检索证据。 +重排序默认关闭。设置 `ENABLE_RERANK=true` 可使用关键词规则；同时设置 `RERANKER_MODEL` 为已下载的 Ollama 生成模型名称，则使用模型评分。它会逐个候选调用模型，增加查询耗时。 -`/search` 的 `stages` 返回 `keyword`、`vector`、`fusion`、`rerank` 和 `retrieval` 五个阶段。每项包含 `status` 与 `latency_ms`；未启用阶段的耗时为 `null`，向量或模型异常时会显示已有的 `fallback:<异常类别>` 状态。控制台在检索结果上方展示这些阶段信息。 +### 扫描 PDF 与容器配置 -### 可选模型重排 +- **OCR**：额外安装 Poppler（`pdftoppm`）、Tesseract 及所需语言包，设置 `OCR_ENABLED=true`。中文扫描资料需选择已安装的中文语言包，例如 `OCR_LANGUAGE=chi_sim+eng`。OCR 仅补充没有提取到文本的页面。 +- **Docker**：仓库提供 [Dockerfile](Dockerfile) 和 [Compose 配置](docker-compose.yml)，组合 API、Milvus、etcd 与 MinIO。Ollama 需单独运行；`LLM_BASE_URL` 必须是 API 容器可访问的地址，容器内的 `localhost` 不指向宿主机。默认 Compose 会发布宿主机端口，运行前应按自己的访问范围调整。 -默认不调用重排模型。需要使用本地 Ollama 对混合召回候选进行相关性评分时，设置： +## 工作方式 -```dotenv -ENABLE_RERANK=true -RERANKER_MODEL=<本地重排模型名> -CANDIDATE_K=20 +```mermaid +flowchart LR + D["PDF / Markdown / TXT"] --> P["解析与分块"] + P --> B["BM25 索引"] + P -. 可选 .-> V["Ollama 向量化 + Milvus"] + Q["问题"] --> R["检索与 RRF 融合"] + B --> R + V -.-> R + R --> K["可选重排序与阈值过滤"] + K --> S["搜索结果"] + K --> G{"问答：有可用片段？"} + G -->|是| A["Ollama 生成 + 来源信息"] + G -->|否| F["refused"] ``` -系统要求模型只返回 `0..1` 的单个数字；模型不可用、超时或输出无法解析时，保留重排前的 RRF 顺序，并在 `retrieval_status.rerank` 中记录 `fallback`。模型重排按候选逐条调用 Ollama，可能增加延迟，建议从较小的 `CANDIDATE_K` 开始。清空 `RERANKER_MODEL` 可回退到不需要模型的确定性关键词重排。 +后端使用 FastAPI；SQLite 保存文档、片段、任务及会话，BM25 索引默认写入 `data/bm25.json`，上传原件保存到 `data/uploads/`。前端是由后端直接托管的 HTML、CSS 和 JavaScript，无需独立的前端构建流程。 -## 测试 +## API 概览 -```bash -pytest -ruff check app tests scripts -python -c "from app.main import app; print(app.title)" -``` +| 接口 | 用途 | +| --- | --- | +| `GET /health` | 服务与依赖状态 | +| `POST /documents/upload` | 以 multipart 字段 `file` 上传文档，返回索引任务 | +| `GET /documents` | 列出资料，支持来源、标签与状态筛选 | +| `PATCH /documents/{document_id}/metadata` | 编辑来源、标签和描述 | +| `POST /documents/reindex` | 提交重建索引任务 | +| `GET /jobs/{job_id}` | 查询任务进度与错误 | +| `POST /jobs/{job_id}/retry` | 重试符合条件的失败任务 | +| `POST /jobs/{job_id}/cancel` | 取消尚未开始运行的排队任务 | +| `POST /search` | 检索片段，支持 `source`、`tag`、`document_ids` 范围筛选 | +| `POST /chat` | 返回回答、来源和工作流状态 | +| `POST /chat/stream` | 通过 SSE 返回阶段、来源和文本事件 | -默认测试不连接真实 Ollama、Milvus 或外部 API。API 测试在安装完整依赖后执行；如果 FastAPI 尚未安装，相关测试会被标记为 skipped，不会伪造通过。 +完整请求与响应结构见运行后的 [API 文档](http://127.0.0.1:8000/docs)。API 问答可传入 `session_id` 保留有限历史；历史用于对话上下文，不作为文档证据。 -## 评估 +## 开发与验证 -评估数据位于 `data/eval_questions.json`，包含 10～20 条问题、期望命中文件/关键内容，以及拒答样例。运行： +安装上述开发依赖后，在仓库根目录运行： ```bash -python scripts/evaluate.py --output data/eval-result.json +ruff check app tests scripts +pytest +python scripts/evaluate.py --retrieval-mode bm25 --output evaluation-bm25.json ``` -默认检索模式是 `bm25`，且重排 `disabled`：只运行本地 BM25 基线，不访问 Ollama、Milvus 或外部 API。可以用同一问题集比较不同检索和重排路径： +BM25 评估不调用模型服务。其他配置可通过 `python scripts/evaluate.py --help` 查看，例如使用已下载的向量化模型比较混合检索： ```bash -# 默认离线基线 -python scripts/evaluate.py --reranker-mode disabled --output /tmp/insight-eval-disabled.json - -# 确定性的关键词重排，不需要模型服务 -python scripts/evaluate.py --reranker-mode keyword --output /tmp/insight-eval-keyword.json - -# 显式启用 Ollama 重排；每个候选片段会产生一次本地模型请求 -python scripts/evaluate.py \ - --reranker-mode ollama \ - --reranker-model "$RERANKER_MODEL" \ - --ollama-base-url "${LLM_BASE_URL:-http://localhost:11434}" \ - --top-k 5 \ - --output /tmp/insight-eval-ollama.json - -# 向量-only：显式调用 Ollama Embedding，使用内存向量后端 -python scripts/evaluate.py \ - --retrieval-mode vector \ - --vector-backend memory \ - --embedding-model "${EMBEDDING_MODEL:-nomic-embed-text}" \ - --ollama-base-url "${LLM_BASE_URL:-http://localhost:11434}" \ - --vector-score-threshold "${VECTOR_SCORE_THRESHOLD:-0.7}" \ - --output /tmp/insight-eval-vector.json - -# 混合检索 + Milvus Lite：URI 和 collection 请按本次实验隔离 python scripts/evaluate.py \ --retrieval-mode hybrid \ - --vector-backend milvus \ - --embedding-model "${EMBEDDING_MODEL:-nomic-embed-text}" \ - --ollama-base-url "${LLM_BASE_URL:-http://localhost:11434}" \ - --milvus-uri /tmp/insight-eval-milvus.db \ - --milvus-collection insight_eval_hybrid \ - --vector-score-threshold "${VECTOR_SCORE_THRESHOLD:-0.7}" \ - --output /tmp/insight-eval-hybrid.json + --vector-backend memory \ + --embedding-model nomic-embed-text \ + --output evaluation-hybrid.json ``` -输出保留 `hit_rate`、`mrr`、`refusal_accuracy` 和 `average_latency_ms`，并新增 `refusal_calibration`（阈值、拒答样例数、误报回答数、误报率）、`retrieval_mode`、`profile`、`models.embedding`、向量后端参数、`average_stage_latency_ms` 以及每条问题的 `stage_status`/`stage_timings_ms`。向量分数会归一化到 `[0, 1]`；`VECTOR_SCORE_THRESHOLD` 默认 `0.7`，在向量/混合检索中于 RRF 前过滤弱候选，`SCORE_THRESHOLD` 仍用于融合结果。可以使用同一问题集重复运行不同阈值，比较 `hit_rate`/`mrr` 与 `refusal_calibration.false_positive_rate`；BM25 默认模式不启用向量阈值。`null` 表示阶段或指标不适用，不代表零毫秒；向量 profile 的模型、地址、后端、URI、collection 和实际延迟会写入结果。`vector`/`hybrid` profile 缺少 Embedding 模型或 Milvus URI 时会快速失败。本项目不预置或声称任何准确率、延迟或模型压缩指标；这些数值必须由本机按指定语料和配置重新测得。 +报告包含命中率、MRR、阶段耗时与拒答相关指标。这里的 `refusal_accuracy` 衡量标注为应拒答的问题是否返回空检索结果，**不衡量生成答案的事实正确性**。内置数据用于回归和演示，不能替代真实业务语料上的人工评估。 + +主要代码入口： + +| 路径 | 内容 | +| --- | --- | +| [`app/api/`](app/api/) | HTTP 路由与接口编排 | +| [`app/ingestion/`](app/ingestion/) | 文档解析、分块与 OCR | +| [`app/retrieval/`](app/retrieval/) | BM25、向量存储与混合检索 | +| [`app/services/`](app/services/) | 资料管理、任务、Ollama、问答和会话 | +| [`app/web/`](app/web/) | 网页控制台 | +| [`scripts/evaluate.py`](scripts/evaluate.py) | 检索评估入口 | +| [`tests/`](tests/) | 自动化测试 | + +## 当前边界 -## 已知限制 +- **来源信息辅助核对**：系统通过提示词要求基于资料回答并引用来源，尚未实现逐条答案与引用的一致性验证。检索分数也不是答案可信度。 +- **检索与问答范围有区别**：`/search` 支持文档范围筛选；当前 `/chat` 和 `/chat/stream` 不应用这些筛选参数。 +- **面向本地使用**：API 未实现用户认证或权限隔离。快速开始仅绑定 `127.0.0.1`；如需共享访问，应先补齐访问控制。配置远程模型地址会将相应查询或文档片段发送至该服务。 +- **格式与业务适用性**：暂不支持 Word 文档。复杂 PDF 和扫描件的解析质量需核对；涉及正式气象业务判断时，应回到有效的原始业务文件复核。 -- PDF 标题识别依赖文档文本层，扫描图片 PDF 需要 OCR 扩展。 -- 扫描 PDF OCR 依赖本机 Poppler、Tesseract 和相应语言包；复杂表格、手写文字和版面结构不保证识别质量。 -- Milvus 集合的向量维度必须与当前 embedding 模型一致，切换模型后需要重建索引。 -- 向量评估需要本地 Ollama Embedding；Milvus Lite 评估会写入指定 URI，重复实验应使用新的 collection 或临时数据库文件。不同 embedding 模型的分数分布可能不同，需要通过 `VECTOR_SCORE_THRESHOLD` 做本地校准；默认值不是准确率保证。 +## 反馈与贡献 -- Web Console 面向本地单用户场景，不提供认证、权限、会话列表或复杂文档管理功能。 -- 索引任务由单进程本地 worker 执行，不提供跨机器任务调度；进程重启后的 running 任务需要重试。 -- 原生流式效果取决于 Ollama 服务和模型；流式连接异常时会退化为完整回答事件。 -- Rerank 目前作为可选 adapter；模型不可用时保留混合检索顺序并记录 fallback。 -- 附带的气象资料为参考样例，不能替代正式气象业务规范。 +项目由 [zhenkun26](https://github.com/zhenkun26) 维护。使用问题、缺陷和功能建议可提交至 [GitHub Issues](https://github.com/zhenkun26/Insight/issues)，代码或文档改进欢迎提交 Pull Request。 -## 路线图 +报告问题时，请附上复现步骤、Python 版本、相关配置项和错误状态，并去除敏感资料与凭据。提交代码改动前，请运行 lint、相关测试和受影响的检索评估。 -- 增强 OCR 和更稳健的章节识别。 -- 增加可选 LangGraph 状态图和节点级追踪。 -- 增加真实公开资料的许可与来源管理。 -- 增加可选的跨编码器 Rerank 和更系统的离线评估集。 +## 许可证 -> 本项目由 Vibe Coding 辅助实现落地。Built with Vibe Coding. +采用 [MIT License](LICENSE)。模型和导入资料分别遵循其自身许可。 From 7f4d4dccb591b2815b0aeca4ba065da1d349567e Mon Sep 17 00:00:00 2001 From: zhenkun Date: Sat, 17 Oct 2026 03:51:00 -0400 Subject: [PATCH 3/5] docs: record permission repair and attribute investigation --- docs/REFACTOR_STATUS.md | 55 ++++++++++++++++++++++++++++++++--------- 1 file changed, 44 insertions(+), 11 deletions(-) diff --git a/docs/REFACTOR_STATUS.md b/docs/REFACTOR_STATUS.md index 5502f25..f7d9826 100644 --- a/docs/REFACTOR_STATUS.md +++ b/docs/REFACTOR_STATUS.md @@ -1,6 +1,6 @@ # Refactoring Status and Resume Point -Updated: 2026-10-02, America/New_York +Updated: 2026-10-17, America/New_York This is the current-state entry for [the refactoring plan](../REFACTOR_INTEGRATION_PLAN.md). User instructions grant authority; this file records that authority and never creates it. @@ -8,16 +8,18 @@ This is the current-state entry for [the refactoring plan](../REFACTOR_INTEGRATI | Field | Current value | | --- | --- | -| Packet | PLAN-00: write and review the refactoring/integration plan | -| Authorized scope | Markdown planning and project recovery records in Insight; workflow consistency review | -| User reference | 2026-10-02 request to proceed one item at a time, using software-engineering and personal-use standards, with refactoring, boundaries, implementation planning, memory and consistency review | -| Planning disposition | COMPLETE | +| Packet | Local maintenance follow-up: record the permission repair and verification conclusions | +| Authorized scope | Update this status file with the completed permission repair, passed checks, historical extended-attribute limitations, and the owner's decision to retain extended attributes | +| User reference | 2026-10-17 instructions to remove only the executable bits from the verified 144 files, preserve other permissions/content/local commits, investigate extended attributes read-only, and record the conclusions | +| Planning disposition | COMPLETE (PLAN-00) | +| Maintenance disposition | COMPLETE; permission repair and investigation closed with the historical attribute limitation accepted | | Implementation disposition | NOT_STARTED | | Active implementation increment | NONE | -| Branch | `codex/refactor-integration-plan` | +| Branch | `codex/permission-repair-status` | | Application baseline | `abb59d7515119e9dcc4ee13bc307cbdc4bed8841` | -| Delivery scope | Three new Markdown files; existing application/specification/source contents remain unchanged | -| Remote delivery | Not authorized; retain the planning commit locally | +| Local main | `47e165fddbf36b0da046bf94c02f9fdf7f9c0b28`, matching `origin/main` at the completed synchronization check | +| Delivery scope | This existing Markdown file only; no application implementation changes | +| Remote delivery | This status update and the planning commit remain local; the separately authorized README-only [PR #1](https://github.com/zhenkun26/Insight/pull/1) was merged into `main` | | Next eligible increment | R01, after the owner instructs work to start | ## Authoritative references @@ -42,12 +44,43 @@ No implementation OpenSpec change exists for this program yet. The roadmap is no | R07 | PLANNED | None; adapter contracts and optional experiments are separate tracks | | R08 | PLANNED | None; engineering and professional acceptance are separate | +## Local permission maintenance — 2026-10-17 + +### Scope and completed repair + +The original 144 application-baseline files had filesystem permissions `0700`, while their Git index entries were `100644` (regular, non-executable files). All 144 file contents matched their indexed blobs. Git displayed the executable-bit differences as `100644 -> 100755`; this did not mean their actual filesystem permissions were `0755`. + +With the owner's explicit authorization, only the executable bits on those 144 verified regular files were removed: `0700 -> 0600`. Their existing owner-only read/write permissions were retained. The other four tracked files remained `0644`. Directories and `.git` were excluded from the permission repair; no recursive permission reset, Git configuration change, file deletion, or commit rewrite was performed. + +The following results describe the completed repair and subsequent read-only investigation, before this documentation-only update. They do not establish application runtime behavior or completion of R01. + +| Check | Result | Evidence and scope | +| --- | --- | --- | +| File contents | PASS | All 148 tracked file contents matched their indexed Git blob hashes; no content mismatch was found | +| Target permissions | PASS | Exactly the verified 144 baseline files were `0600`; the other four tracked files remained `0644` | +| Other file metadata | PASS | The repair checker confirmed the expected mode change while preserving recorded ownership, non-executable permission bits, content, modification time, file flags, and the other recorded metadata fields; metadata-change time was not expected to remain unchanged | +| ACL inspection | PASS | No extended ACL entries were present on the 144 repaired files at the final read-only check | +| Git state | PASS | Index entries, local branch references, and the checked-out branch were preserved; the worktree was clean after repair | +| Local commits | PASS | Planning commit `e0406a9e4b4f000f6829e73470e2bf07bfcabbed` remained on `codex/refactor-integration-plan`; the local README commit `43b4136f52551fd272d91dea7ef8830b31af2668` and local `main` at `47e165fddbf36b0da046bf94c02f9fdf7f9c0b28` were preserved | +| Git object integrity | PASS | `git fsck --full --no-reflogs --no-dangling` completed successfully | +| Current extended-attribute stability | PASS | Native read-only attribute enumeration covered 1,114 files/directories, including `.git`; only `com.apple.provenance` was found, with no differences across repeated reads and the investigation's start/end snapshots | +| Original aggregate extended-attribute comparison | FAIL | The repair checker found different raw `xattr -lx` command outputs before and after repair; it did not save those outputs or emit a per-file/per-attribute difference | +| Exact historical attribute reconstruction | BLOCKED | No retained pre-repair attribute values or available local snapshots could restore the precise historical differences; affected paths and old/new bytes cannot be listed reliably | + +### Explanation, impact, and accepted disposition + +The strongly supported explanation for the aggregate comparison failure is a macOS provenance update triggered by setting file permissions. The repaired files' current `com.apple.provenance` values match the executing ChatGPT/Codex process lineage, and system logs from the repair time contain the same provenance marker. [FFRI's original research](https://github.com/FFRI/ShowProvenanceInfo) identifies `setmode` among operations tracked by the Provenance Sandbox. This supports the explanation but does not reconstruct the missing historical attribute values or prove that every repaired file's attribute changed. + +No file-content damage, unexpected final permission/ACL state, or Git integrity problem was found. This is not a claim that every extended attribute remained unchanged. Full application execution, writes from other applications, and all possible macOS security-policy effects were not tested during the read-only investigation. + +The owner accepted retaining the current extended attributes and requested no further attribute handling. Keep `com.apple.provenance` intact; do not clear, rewrite, or attempt to restore it from inferred values. Permission repair and the investigation are closed with the historical reconstruction limitation recorded. Application implementation remains `NOT_STARTED`, active increment remains `NONE`, and R01 still requires a separate instruction to start. + ## Known constraints and verification gaps - OpenSpec CLI is unavailable in the current shell; formal validation is BLOCKED. No CLI was installed in PLAN-00. - The ordinary local Python environment lacks required application/test dependencies. PLAN-00 is a documentation-only delivery and does not claim application tests ran on it. - Prior syntax/lint, pure-memory tests and GitHub CI apply to the investigation/application baseline. They do not qualify future refactoring code, adapters or professional correctness. -- Existing 144 file-mode differences are unrelated work. They must not enter this packet's staged diff or later refactoring commits accidentally. +- The former 144 file-mode differences were resolved by the owner-authorized repair recorded above. Preserve the resulting `0600` file permissions and retained extended attributes; do not reintroduce executable bits or clear system metadata during later work. - Real Ollama/Milvus, complex PDF parsing and cancellation propagation require later real checks. - There is no established real meteorology corpus or annotated domain test set. The twelve Atmos-RAG seeds are not formal evaluation data. - Filesystem deletion is prohibited, including temporary/test cleanup. A usable retained-artifact strategy must be established before affected tests run. @@ -63,10 +96,10 @@ No implementation OpenSpec change exists for this program yet. The roadmap is no ## Locating the planning delivery commit -The delivery commit is identified in Git history rather than copied into a self-referential tracking commit: +The original PLAN-00 delivery was commit `e0406a9e4b4f000f6829e73470e2bf07bfcabbed` on `codex/refactor-integration-plan`, containing the plan, this status file, and the consistency review. Later maintenance updates do not change that planning delivery or authorize implementation. To inspect its record: ```sh -git log -1 --format='%H %s' -- REFACTOR_INTEGRATION_PLAN.md docs/REFACTOR_STATUS.md docs/reviews/2026-10-02-refactor-plan-consistency.md +git show --stat e0406a9e4b4f000f6829e73470e2bf07bfcabbed ``` At later increments, record existing relevant commit references and receipt links. Do not create an additional commit solely to record a document's own new hash. From fb3ff3e536d5f22031203854c09f4bcf4d1eb31c Mon Sep 17 00:00:00 2001 From: zhenkun Date: Sat, 17 Oct 2026 03:54:24 -0400 Subject: [PATCH 4/5] docs: record authorized planning branch delivery --- docs/REFACTOR_STATUS.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/REFACTOR_STATUS.md b/docs/REFACTOR_STATUS.md index f7d9826..4a19c16 100644 --- a/docs/REFACTOR_STATUS.md +++ b/docs/REFACTOR_STATUS.md @@ -18,8 +18,8 @@ This is the current-state entry for [the refactoring plan](../REFACTOR_INTEGRATI | Branch | `codex/permission-repair-status` | | Application baseline | `abb59d7515119e9dcc4ee13bc307cbdc4bed8841` | | Local main | `47e165fddbf36b0da046bf94c02f9fdf7f9c0b28`, matching `origin/main` at the completed synchronization check | -| Delivery scope | This existing Markdown file only; no application implementation changes | -| Remote delivery | This status update and the planning commit remain local; the separately authorized README-only [PR #1](https://github.com/zhenkun26/Insight/pull/1) was merged into `main` | +| Delivery scope | This status-record increment changes this existing Markdown file only. Initial remote delivery also includes the existing refactoring plan and planning consistency review; there are no application implementation changes | +| Remote delivery | Owner-authorized on 2026-10-17: push the status update and its planning-document dependencies to `origin/codex/permission-repair-status`; do not merge into `main`. Remote publication is verified in the delivery response. The separately authorized README-only [PR #1](https://github.com/zhenkun26/Insight/pull/1) was already merged into `main` | | Next eligible increment | R01, after the owner instructs work to start | ## Authoritative references From 268e32fd3751477951dc776c6c245ae5befc515d Mon Sep 17 00:00:00 2001 From: zhenkun Date: Sat, 17 Oct 2026 04:16:35 -0400 Subject: [PATCH 5/5] docs: identify draft delivery and maintenance evidence --- docs/REFACTOR_STATUS.md | 49 ++++++++++++++++++++++++++++------------- 1 file changed, 34 insertions(+), 15 deletions(-) diff --git a/docs/REFACTOR_STATUS.md b/docs/REFACTOR_STATUS.md index aa4fe92..cee4323 100644 --- a/docs/REFACTOR_STATUS.md +++ b/docs/REFACTOR_STATUS.md @@ -8,9 +8,9 @@ This is the current-state entry for [the refactoring plan](../REFACTOR_INTEGRATI | Field | Current value | | --- | --- | -| Packet | Local maintenance follow-up: record the permission repair and verification conclusions | -| Authorized scope | Update this status file with the completed permission repair, passed checks, historical extended-attribute limitations, and the owner's decision to retain extended attributes | -| User reference | 2026-10-17 instructions to remove only the executable bits from the verified 144 files, preserve other permissions/content/local commits, investigate extended attributes read-only, and record the conclusions | +| Packet | P3 review follow-up: identify actual PR delivery and historical maintenance evidence | +| Authorized scope | Update this status file only, verify, commit, and push to the current branch; keep PR #2 as a draft and do not merge | +| User reference | 2026-10-17 permission-repair and read-only investigation instructions, followed by approval to address the two P3 review findings with truthful evidence identities and unrecoverable-information limits | | Planning disposition | COMPLETE (PLAN-00) | | Maintenance disposition | COMPLETE; permission repair and investigation closed with the historical attribute limitation accepted | | Implementation disposition | NOT_STARTED | @@ -19,8 +19,9 @@ This is the current-state entry for [the refactoring plan](../REFACTOR_INTEGRATI | Application baseline | `abb59d7515119e9dcc4ee13bc307cbdc4bed8841` | | Local main | `47e165fddbf36b0da046bf94c02f9fdf7f9c0b28`, matching `origin/main` at the completed synchronization check | | Delivery scope | This status-record increment changes this existing Markdown file only. Initial remote delivery also includes the existing refactoring plan and planning consistency review; there are no application implementation changes | -| Remote delivery | Owner-authorized on 2026-10-17: push the status update and its planning-document dependencies to `origin/codex/permission-repair-status`; do not merge into `main`. Remote publication is verified in the delivery response. The separately authorized README-only [PR #1](https://github.com/zhenkun26/Insight/pull/1) was already merged into `main` | -| Review disposition | Owner-authorized Draft PR from `codex/permission-repair-status` to `main`, limited to the plan, this status record, and the planning consistency review; retain draft state and do not merge. The source branch incorporates the existing `main` history to exclude already-merged README changes from the PR comparison | +| Remote delivery | PUBLISHED: the plan, status record, and consistency review are on [origin/codex/permission-repair-status](https://github.com/zhenkun26/Insight/tree/codex/permission-repair-status). The verified pre-follow-up remote head is `a095d67821291fb6a7398133e680285687befaf8`; this documentation follow-up is authorized for the same branch | +| Review disposition | [PR #2](https://github.com/zhenkun26/Insight/pull/2) exists and is OPEN / DRAFT, from `codex/permission-repair-status` to `main`; it is not merged. Its verified comparison contains exactly the plan, this status record, and the planning consistency review. Existing `main` history is included in the source branch, excluding already-merged README changes. Keep the PR as a draft; do not merge | +| CI evidence checkpoint | Both the [push run](https://github.com/zhenkun26/Insight/actions/runs/39016103272) and [PR run](https://github.com/zhenkun26/Insight/actions/runs/39016120920) passed for `a095d67821291fb6a7398133e680285687befaf8` on 2026-10-17. These results apply only to that revision; subsequent commits need their own results in [PR #2 checks](https://github.com/zhenkun26/Insight/pull/2/checks). CI does not verify this workstation's permissions or historical attributes | | Next eligible increment | R01, after the owner instructs work to start | ## Authoritative references @@ -55,18 +56,36 @@ With the owner's explicit authorization, only the executable bits on those 144 v The following results describe the completed repair and subsequent read-only investigation, before this documentation-only update. They do not establish application runtime behavior or completion of R01. +### Historical verification identities and retention + +The maintenance baseline was HEAD `43b4136f52551fd272d91dea7ef8830b31af2668` on `codex/readme-bilingual`, with 148 tracked files. The exact 144-file target set is the tracked path set at `abb59d7515119e9dcc4ee13bc307cbdc4bed8841`, reproducible with `git ls-tree -r --name-only abb59d7515119e9dcc4ee13bc307cbdc4bed8841`. The index-entry comparison used SHA-256 `7ebc7eccaa963d72c47c8c1ac2de48b87cd51c7accff675028166e9b2156ab9c` of `git ls-files --stage -z` output. + +Source: local session `01a148a2-417c-7b60-abe2-b81528cabe15`, containing the checker invocations and tool-result summaries. It is not committed or available to repository-only readers; availability depends on local session retention. The evidence IDs below name those historical records; they are not newly executed checks. All were performed by the same authoring agent, without independent review. Times are recorded tool-result times on 2026-10-17 in America/New_York (UTC-04:00), rounded down to seconds, not exact filesystem syscall times. + +| Evidence ID | Recorded result time | Identifiable method and result | +| --- | --- | --- | +| PERM-01 | 03:42:54 | Preflight output: 144 targets at `0700`, all matching indexed blob hashes; explicit HEAD and planning-commit identities | +| PERM-02 | 03:43:41 | Repair used `os.fchmod(fd, mode & ~0o111)` after identity/mode checks. Its in-memory stat/content comparison passed before the raw aggregate `xattr -lx` comparison raised `AssertionError: Extended attributes changed` | +| PERM-03 | 03:44:39 | Separate post-repair checker by the same agent: 148 contents matched their indexed blobs, 144 targets were `0600`, four other files were `0644`, the index-entry digest and expected local branch tips matched, and the worktree was clean | +| XATTR-01 | 03:45:49 through 03:48:14 | Native `listxattr`/`getxattr` reads compared attribute names, lengths, and value hashes across 1,114 paths; final output reported no attribute differences, no target ACL entries, no content/mode mismatches, and successful `git fsck --full --no-reflogs --no-dangling` | +| XATTR-02 | 03:47:23 | Read-only comparison of file/process provenance markers and the system-log window 03:43:35–03:43:50 supported the provenance-update explanation; this was not a reconstruction of pre-repair attribute values | + +Retention limit: the local transcript identifies the methods and summaries, but no standalone, portable maintenance receipt or full per-file stat manifest was archived. The native attribute snapshots were held in execution/session state; no complete exported snapshot pair is available in this repository. The pre/post raw attribute outputs from PERM-02 were neither retained nor printed as a per-file difference. This document is a traceable summary of those local observations, not an independent reproduction of their historical before/after state. The 1,114-path count belongs to XATTR-01 and is not a fixed count for later repository states. + +### Historical check results + | Check | Result | Evidence and scope | | --- | --- | --- | -| File contents | PASS | All 148 tracked file contents matched their indexed Git blob hashes; no content mismatch was found | -| Target permissions | PASS | Exactly the verified 144 baseline files were `0600`; the other four tracked files remained `0644` | -| Other file metadata | PASS | The repair checker confirmed the expected mode change while preserving recorded ownership, non-executable permission bits, content, modification time, file flags, and the other recorded metadata fields; metadata-change time was not expected to remain unchanged | -| ACL inspection | PASS | No extended ACL entries were present on the 144 repaired files at the final read-only check | -| Git state | PASS | Index entries, local branch references, and the checked-out branch were preserved; the worktree was clean after repair | -| Local commits | PASS | Planning commit `e0406a9e4b4f000f6829e73470e2bf07bfcabbed` remained on `codex/refactor-integration-plan`; the local README commit `43b4136f52551fd272d91dea7ef8830b31af2668` and local `main` at `47e165fddbf36b0da046bf94c02f9fdf7f9c0b28` were preserved | -| Git object integrity | PASS | `git fsck --full --no-reflogs --no-dangling` completed successfully | -| Current extended-attribute stability | PASS | Native read-only attribute enumeration covered 1,114 files/directories, including `.git`; only `com.apple.provenance` was found, with no differences across repeated reads and the investigation's start/end snapshots | -| Original aggregate extended-attribute comparison | FAIL | The repair checker found different raw `xattr -lx` command outputs before and after repair; it did not save those outputs or emit a per-file/per-attribute difference | -| Exact historical attribute reconstruction | BLOCKED | No retained pre-repair attribute values or available local snapshots could restore the precise historical differences; affected paths and old/new bytes cannot be listed reliably | +| File contents | PASS | PERM-03 / XATTR-01: all 148 tracked file contents matched their indexed Git blob hashes; no content mismatch was found | +| Target permissions | PASS | PERM-03 / XATTR-01: exactly the verified 144 baseline files were `0600`; the other four tracked files remained `0644` | +| Recorded stat fields | PASS | PERM-02: the in-memory comparison preserved recorded ownership, non-executable permission bits, content, modification time, file flags, and the other stat fields. Metadata-change time was excluded; the full original manifest was not archived | +| ACL inspection | PASS | XATTR-01: no extended ACL entries were present on the 144 repaired files at the final read-only check | +| Git state | PASS | PERM-03 / XATTR-01: index entries, local branch references, and the checked-out branch were preserved; the worktree was clean after repair | +| Local commits | PASS | PERM-01 / PERM-03: planning commit `e0406a9e4b4f000f6829e73470e2bf07bfcabbed` remained on `codex/refactor-integration-plan`; the local README commit `43b4136f52551fd272d91dea7ef8830b31af2668` and local `main` at `47e165fddbf36b0da046bf94c02f9fdf7f9c0b28` were preserved | +| Git object integrity | PASS | XATTR-01: `git fsck --full --no-reflogs --no-dangling` completed successfully | +| Extended-attribute stability during the investigation | PASS | XATTR-01: only `com.apple.provenance` was found across 1,114 files/directories, including `.git`, with no differences across repeated reads and the investigation's start/end snapshots | +| Original aggregate extended-attribute comparison | FAIL | PERM-02: raw `xattr -lx` command outputs differed before and after repair; no saved outputs or per-file/per-attribute difference remain | +| Exact historical attribute reconstruction | BLOCKED | PERM-02 / XATTR-01: no retained pre-repair values or usable local snapshots were found that could restore the differences; affected paths and old/new bytes cannot be listed reliably | ### Explanation, impact, and accepted disposition