Add document RAG: import, search, streamed chat, review, and embeddable widget by bihius · Pull Request #1 · bihius/grounded

bihius · 2026-09-18T09:21:39Z

main still has the Docker/Laravel skeleton. This branch is the rest of the application: a local RAG assistant that indexes documents, answers from retrieved chunks, and says so when the knowledge base does not cover the question.

What landed

Ingestion. Markdown, text, PDF, and URL import (Symfony DomCrawler). Uploads go onto a Redis queue; a worker chunks the text, embeds each chunk with bge-m3, and stores vectors in pgvector.
Retrieval and chat. Cosine search over chunks, a distance threshold on the closest hit (RAG_MAX_DISTANCE=0.6 for bge-m3), then an answer from qwen3:8b. Chat streams over SSE and returns the fragments it used. Below the threshold the model is not called.
Review loop. Thumbs on answers, a needs_review queue, and a resolve action that turns a written answer into a new indexed document.
Interface. Chat, documents, questions, and analytics on their own URLs. Analytics groups similar questions when embeddings exist.
API and widget. REST surface documented in docs/openapi.yaml. /widget.js is a small TypeScript bundle in a shadow root, plus /widget-demo as a host page that is not the app itself.

Fixes on top of that

Tests no longer wipe the development database: phpunit.xml overrides container env vars through <server force="true">.
Creating a document through POST /api/documents now queues indexing, same as file and URL import.
Import endpoints return 201. Indexed documents report completed.
The inbox watcher has a content_hash column to skip duplicates.
Redis retry_after is 180s so it stays above the worker timeout of 120s.
Widget source titles are rendered as text, not HTML. CORS is configured for /api/*. CI uses PHP 8.4.

How to try it

cp .env.example .env   # OLLAMA_URL should be http://host.docker.internal:11434
ollama pull bge-m3 && ollama pull qwen3:8b
docker compose up -d --build
docker compose exec app php artisan key:generate
docker compose exec app php artisan migrate
docker compose restart worker

Then open http://localhost:8000 — import a short note, wait until its status is completed, and ask something it actually contains. Also hit a question it cannot know, rate a bad answer, resolve it from /questions, and load /widget-demo.

docker compose exec app php artisan test

The 0.4 cosine distance cutoff was tuned too tight for bge-m3: chunks that answer the question sit at 0.45-0.52, so every hit was filtered out and the assistant claimed the knowledge base held nothing. Borderline cases were worse — the filter silently trimmed the source list to one or two entries. The threshold now decides only whether the question is covered at all, by looking at the closest hit; the rest of the retrieved chunks stay in the context and in the sources. Raise the default to 0.6 to match the model.

docker-compose exports APP_ENV, DB_CONNECTION, SESSION_DRIVER, QUEUE_CONNECTION and four more as real environment variables. PHPUnit's <env> does not override those, so the suite ran against the development Postgres, Redis and database sessions: RefreshDatabase wiped the working knowledge base on every run, and the smoke test 500'd on a missing sessions table. Switch the whole <php> block to <server force="true">, which does reach the variables Laravel reads. The smoke test no longer needs RefreshDatabase — the route renders a static view — and now asserts the React root is present.

The /api/documents/import-url endpoint had no way to reach it from the interface. Add a URL field next to the file upload, reusing the optional title input and the isUploading state so the two forms cannot fire at once. Importing queues a chunking job, so the panel now refreshes the document list every three seconds — without it a freshly added page sits at "queued" until a manual reload. Source excerpts render through MarkdownMessage as well, matching how the answer itself is displayed.

The single App component held the chat, the document panel and the review panel at once, so every visit paid for all three and the review questions sat under the chat regardless of what the user came for. Each panel now owns its own URL (/, /documents, /questions) and its own file; App is only the shell that picks one by pathname and renders the nav. All three routes serve the same Blade view, so no router dependency is needed. The document panel also gains a separate title field for the URL import - previously the file title input was shared - and separate busy flags, so one form no longer blocks the other.

Stage 10 of the plan asked for a REST API with OpenAPI docs plus a widget to embed; only the widget was missing. A host page now gets the assistant from one line: <script src="https://grounded.example/widget.js" defer></script> The widget is plain TypeScript in a shadow root, not the React bundle. The application bundle weighs ~290 kB and carries global Tailwind styles that would collide with whatever page hosts it; this one is 4.5 kB and is isolated from the host's CSS in both directions. It builds separately (`npm run build:widget`) because an embedded script needs a stable URL rather than a hashed name from the Vite manifest, and public/widget.js is committed so the snippet works on a fresh clone. Running on a foreign origin means the chat endpoint has to answer cross-origin requests. Laravel's defaults already allow that, so the test pins the behaviour rather than changing it. /widget-demo serves a page that is deliberately not the application, so the widget can be checked where it actually runs.

GET /api/analytics/questions had no interface, so the numbers it reports were only reachable with curl. The endpoint is untouched; this is the view on top of it. /analytics gets its own page and nav tab, matching the split the other panels just went through: summary tiles, the most asked questions with their counts, and the embedding-based groups of questions that mean the same thing. Questions awaiting review are listed in short, with a link to the panel that already resolves them rather than a second copy of it.

Copilot

🟡 Changes recommended

Unresolved critical and moderate findings include security, deployment, API, and indexing blockers.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Builds the RAG assistant through stage 10, covering ingestion, vector search, streamed chat, review workflows, analytics, React pages, and an embeddable widget.

Changes:

Adds document imports, chunking, embeddings, retrieval, and SSE chat.
Adds feedback, review, analytics, REST/OpenAPI support, and React pages.
Adds Docker queue services, widget bundling, documentation, and tests.

File summaries

File	Change / final review note
`vite.widget.config.js`	Configures standalone widget output.
`vite.config.js`	Enables React compilation.
`tsconfig.json`	Adds TypeScript configuration.
`tests/Unit/SimilarChunkSearchTest.php`	Tests vector similarity search.
`tests/Unit/QuestionAnswererTest.php`	Tests answering and distance thresholds.
`tests/Unit/EmbeddingGeneratorTest.php`	Tests Ollama embeddings.
`tests/Unit/ChunkDocumentTest.php`	Tests chunk embedding jobs; the unsaved document setup fails before the mocked path (moderate, 1 vote).
`tests/Feature/WidgetTest.php`	Tests widget publishing and CORS.
`tests/Feature/QuestionReviewTest.php`	Tests review and analytics APIs.
`tests/Feature/QuestionFeedbackTest.php`	Tests answer ratings.
`tests/Feature/ExampleTest.php`	Tests application pages.
`tests/Feature/DocumentImportTest.php`	Tests Markdown, PDF, and web imports.
`tests/Feature/DocumentChunkingTest.php`	Tests overlapping chunks.
`routes/web.php`	Adds dashboard and widget routes.
`routes/api.php`	Adds APIs; CORS preflight handling is missing (critical, 2 votes), and management routes lack authentication/authorization (critical, 3 votes).
`resources/views/widget-demo.blade.php`	Provides the external widget demo page.
`resources/views/welcome.blade.php`	Provides the React shell; unconditional `@vite` fails without a manifest in clean checkouts (critical, 3 votes).
`resources/js/widget.ts`	Implements the shadow-root widget; interpolating `source.title` into HTML enables XSS (critical, 3 votes).
`resources/js/types.ts`	Defines frontend data types.
`resources/js/pages/ReviewPage.tsx`	Implements the review workflow.
`resources/js/pages/DocumentsPage.tsx`	Implements document management; PDF files cannot be selected (moderate, 1 vote).
`resources/js/pages/ChatPage.tsx`	Implements streamed chat UI.
`resources/js/pages/AnalyticsPage.tsx`	Implements the analytics UI.
`resources/js/components/MarkdownMessage.tsx`	Renders message formatting.
`resources/js/App.tsx`	Selects pages by pathname.
`resources/js/app.js`	Mounts the React application.
`resources/css/app.css`	Includes TypeScript and TSX sources for Tailwind.
`README.md`	Documents widget usage and setup.
`public/widget.js`	Published bundle retains the unsafe `innerHTML` interpolation (critical, 2 votes); regenerate it from the safe source.
`phpunit.xml`	Isolates test environment variables.
`package.json`	Adds React, TypeScript, and widget tooling.
`docs/openapi.yaml`	Documents the REST API; `/search` declares `excerpt` while results provide `content` (moderate, 1 vote).
`Dockerfile`	Adds Redis support; the image lacks the DOM extension required by DomCrawler (critical, 1 vote).
`docker-compose.yml`	Adds application and worker services; the 90-second Redis retry window is shorter than the 120-second worker timeout (critical, 1 vote).
`database/migrations/2026_09_18_000002_add_embedding_to_questions_table.php`	Adds question vectors.
`database/migrations/2026_09_18_000001_create_questions_table.php`	Creates question and feedback storage.
`database/migrations/2026_09_17_100000_add_indexing_fields_to_documents_table.php`	Adds indexing status; the store path does not dispatch chunking, leaving text documents unsearchable (critical, 1 vote).
`database/migrations/2026_09_17_000001_add_embedding_to_document_chunks_table.php`	Adds chunk embeddings.
`config/services.php`	Configures Ollama and RAG settings.
`composer.lock`	Locks document parsing dependencies.
`composer.json`	Adds PDF and HTML parsing packages.
`app/Services/SimilarChunkSearch.php`	Performs vector similarity search.
`app/Services/QuestionAnswerer.php`	Generates regular and streamed answers; the SSE path lacks coverage for framing, completion, sources, and no-knowledge behavior (nit, 1 vote).
`app/Services/QuestionAnalytics.php`	Builds analytics; each request synchronously embeds all missing question vectors, risking growth and timeouts (moderate, 3 votes).
`app/Services/EmbeddingGenerator.php`	Calls Ollama embeddings.
`app/Models/Question.php`	Defines question persistence.
`app/Models/Document.php`	Extends document model handling.
`app/Jobs/ChunkDocument.php`	Chunks and embeds documents; records undocumented `ready` status instead of `completed` (moderate, 2 votes).
`app/Http/Controllers/QuestionController.php`	Handles feedback and review APIs.
`app/Http/Controllers/DocumentController.php`	URL imports allow SSRF (critical, 3 votes); imports return 200 instead of the required 201 (moderate, 3 votes); search results use `content` instead of documented `excerpt` (moderate, 1 vote).
`app/Http/Controllers/ChatController.php`	Exposes chat endpoints.
`app/Console/Commands/WatchDocuments.php`	Watches the inbox; `content_hash` is absent from the schema and fillable fields, causing import failure (critical, 3 votes).
`.gitignore`	Ignores fixtures and PHPUnit artifacts.
`.env.example`	Adds service settings; documented `localhost:11434` is unreachable from the application and worker containers (critical, 3 votes).

Review details

Suppressed comments (6)

app/Http/Controllers/DocumentController.php:85

As with importUrl(), returning the model directly produces 200 OK, while the PDF/Markdown import tests require 201 Created. Wrap this response with an explicit 201 status.

        return $document;

app/Http/Controllers/DocumentController.php:28

The /search endpoint returns the raw SimilarChunkSearch rows, whose field is content, but the documented Source response and the shared frontend type require excerpt. API consumers therefore receive a different shape from the contract; map this endpoint to the source DTO/alias without breaking the content field still used by QuestionAnswerer.

        return $search->search($data['q'], $data['limit'] ?? 5);

app/Services/QuestionAnswerer.php:20

The new SSE path is not exercised by the tests, while the non-streaming answer() path is. Without a stream test, token framing, the terminal done event, source serialization, and the no-knowledge branch can regress without detection; add an endpoint/service test that consumes the streamed response.

    public function stream(string $question, int $limit = 5): StreamedResponse
    {
        $questionRecord = Question::create(['question' => $question]);
        $chunks = $this->relevantChunks($question, $limit);

        return response()->stream(function () use ($question, $questionRecord, $chunks): void {

docs/openapi.yaml:250

SimilarChunkSearch selects content, position, document_id, title, and distance, but the documented /search response references Source, which requires excerpt and does not define content. Clients generated from this OpenAPI document will receive a shape that does not satisfy the declared schema; either map the endpoint to Source or document its actual result type.

                type: array
                items:
                  $ref: '#/components/schemas/Source'

resources/js/pages/DocumentsPage.tsx:97

The page cannot select PDF files even though the importer validates and tests PDF support, so PDF import is inaccessible through the UI. Include .pdf in the file input's accepted extensions.
tests/Unit/ChunkDocumentTest.php:17
This test gives the job an unsaved Document, but ChunkDocument::handle() immediately calls update() on it. Since the test has no RefreshDatabase setup and the model lacks required document fields, it will try to persist an invalid/nonexistent row before reaching the mocked embedding path; persist a valid document with the database reset trait or mock the document update.

Files reviewed: 51/56 changed files
Comments generated: 14
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+REDIS_HOST=redis
 REDIS_PASSWORD=null
 REDIS_PORT=6379
+OLLAMA_URL=http://localhost:11434


+    && apt-get install -y --no-install-recommends libpq-dev $PHPIZE_DEPS \
    && docker-php-ext-install pdo_pgsql \


+                : file_get_contents($path);
+            $hash = hash('sha256', $content);
+
+            if (Document::where('content_hash', $hash)->exists()) {


+        $data = $request->validate([
+            'url' => ['required', 'url'],
+            'title' => ['sometimes', 'string', 'max:255'],
+        ]);
+        $html = Http::timeout(20)->get($data['url'])->throw()->body();


+    public function up(): void
+    {
+        Schema::table('documents', function (Blueprint $table) {
+            $table->string('indexing_status')->default('queued');


 Route::get('/documents', [DocumentController::class, 'index']);
+Route::get('/search', [DocumentController::class, 'search']);
+Route::post('/chat', ChatController::class);
+Route::post('/chat/stream', [ChatController::class, 'stream']);


 Route::get('/documents', [DocumentController::class, 'index']);
+Route::get('/search', [DocumentController::class, 'search']);
+Route::post('/chat', ChatController::class);
+Route::post('/chat/stream', [ChatController::class, 'stream']);
+Route::get('/analytics/questions', [QuestionController::class, 'analytics']);
+Route::get('/questions', [QuestionController::class, 'index']);
+Route::post('/questions/{question}/feedback', [QuestionController::class, 'feedback']);
+Route::post('/questions/{question}/resolve', [QuestionController::class, 'resolve']);


+        $this->generateMissingEmbeddings();
+        $questions = DB::table('questions')


The previous README described the project in the present tense regardless of whether something was implemented, and its setup instructions did not work: a fresh checkout fails three times before it serves a page, because the image has no Composer, git does not track storage/framework, and the Blade view needs a Vite manifest. Every command here was executed against a clean export of HEAD in a separate Compose project, and every curl example is a real request with its real response. Limitations are stated rather than implied: character-based chunking, no vector index, no authentication, a documents:watch command that queries a column no migration creates, and a CI workflow whose test step is commented out. Plans live under Roadmap, marked as not implemented.

Imported documents now queue for chunking, the widget renders source titles as text, and CI installs PHP 8.4 so Composer can resolve the lock file.

bihius added 30 commits September 17, 2026 11:20

add document import and Redis queue

828e6cf

add document chunk embeddings column

4b15956

add Ollama embedding service

5b21eb6

generate embeddings in chunking job

a7c3cf7

add semantic chunk search

23c7981

add RAG question answering

310f32c

move container settings to environment files

085f8b0

add RAG sources and confidence threshold

830f907

ignore private ETF notes

4a54496

add streaming chat endpoint

00a86a8

add react frontend shell

e612815

build streaming chat interface

4ed7ab7

add document indexing fields

57dd433

track document import status

95778f4

add document indexing workflow

73a8b3d

add question feedback model

8e9aecd

record questions for feedback

7cc879e

add question feedback endpoint

1669d33

include question id in chat stream

9272faa

import documents from web pages

c717c2d

add chat feedback buttons

df88fcb

add question review workflow

6f80bf4

add question review panel

84c440c

document REST API with OpenAPI

cd87904

add question analytics and semantic groups

b9969c0

translate OpenAPI documentation to English

54a4fea

bihius added 2 commits September 18, 2026 11:18

Copilot AI lite review requested due to automatic review settings September 18, 2026 09:21

Copilot started reviewing on behalf of bihius September 18, 2026 09:22

View session

Copilot AI reviewed Sep 18, 2026

View reviewed changes

bihius added 2 commits September 18, 2026 11:34

Fix review findings that blocked indexing, CORS, and the widget.

96c0ba0

Imported documents now queue for chunking, the widget renders source titles as text, and CI installs PHP 8.4 so Composer can resolve the lock file.

bihius changed the title ~~Build the RAG assistant through stage 10~~ Add document RAG: import, search, streamed chat, review, and embeddable widget Sep 18, 2026

bihius merged commit c253a2e into main Sep 18, 2026
2 checks passed

Navigation Menu

Repository navigation

Add document RAG: import, search, streamed chat, review, and embeddable widget - #1

Add document RAG: import, search, streamed chat, review, and embeddable widget#1
bihius merged 34 commits into
mainfrom
fix/rag-threshold-and-test-isolation

bihius commented Sep 18, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		&& apt-get install -y --no-install-recommends libpq-dev $PHPIZE_DEPS \
		&& docker-php-ext-install pdo_pgsql \

		$this->generateMissingEmbeddings();
		$questions = DB::table('questions')

Conversation

bihius commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What landed

Fixes on top of that

How to try it

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (6)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bihius commented Sep 18, 2026 •

edited

Loading