Genie notes: ignore the speaker tag when searching, soften the unknow…

docs/lld/06-frontend.md

@@ -137,7 +137,7 @@ The widget is TypeScript bundled with microbundle into `dist/index.umd.js` and s
 - `submitOnKeydown` on desktop, and `openOnLoad` only on `/practice`. Elsewhere Genie opens from the app bar, the floating button or the error note (LLD 07).
 - `closeOnOutsideClick = false`: the Genie panel is docked and non-modal (LLD 07).
 
-The system message includes the current editor content, so the assistant can answer "debug my code". A reply that used the site's notes shows a "Used OpenREPL notes" line with a note card, and one that did not says so (LLD 07 section 3a; `parseContextHeader`, `appendContextLine`, styles `.chat-widget__notes` in `widget.css`). Code blocks in replies get **Insert** and **Replace** buttons that call `window.insertcodesnippet` and `window.replacecodesnippet`, which edit the Ace buffer. The model and effort are chosen with a chip in the composer and come from `js/model-choice.js` (default Luna, Low; Gemma 4 31B through OpenRouter when the server has a key, and an unavailable card with Try again and Switch to GPT-6 Luna when it cannot answer); the context sent is the editor code and the terminal's recent output (LLD 07).
+The system message includes the current editor content, so the assistant can answer "debug my code". `NUM_MANDATORY_ENTRIES`, the number of messages at the start of the history that are never trimmed (the last is the editor snapshot that each user message refreshes), is set in `init()` from what was added, not fixed: it was 4 while two of them were the page text and the documentation list, and has to follow the list. A reply that used the site's notes shows a "Used OpenREPL notes" line with a note card, and one that did not says so (LLD 07 section 3a; `parseContextHeader`, `appendContextLine`, styles `.chat-widget__notes` in `widget.css`). Code blocks in replies get **Insert** and **Replace** buttons that call `window.insertcodesnippet` and `window.replacecodesnippet`, which edit the Ace buffer. The model and effort are chosen with a chip in the composer and come from `js/model-choice.js` (default Luna, Low; Gemma 4 31B through OpenRouter when the server has a key, and an unavailable card with Try again and Switch to GPT-6 Luna when it cannot answer); the context sent is the editor code and the terminal's recent output (LLD 07).
 
 ## 6. Practice mode
 

docs/lld/07-ai-features.md

@@ -63,9 +63,9 @@ A request from the Genie panel or from the blog editor can ask the proxy to add
 
 - **The sources.** Notes: Markdown files in `src/resources/knowledge/`, one topic each, compiled into the binary (`bindata/knowledge`, read with `AssetDir("knowledge")`; `Makefile` target `bindata/knowledge`). A note has a header with `title`, `keywords` and an optional `link`, then 80 to 220 words of text; its id is the file name. A note that fails its checks (no header, title or text, an id that is not letters, digits and dashes, a `link` that is not a path on this site: no scheme, no host, no `//`, no backslash or space) is logged and skipped. Blog posts: read with `FetchBlogDataMap`, their HTML turned into text and cut into passages of about 140 words (at most 24 per post), each with the post's title and link `/blog?name=...`. The post's description opens its first passage. Posts are not copied anywhere; the index is rebuilt when a post is saved or deleted (`knowledgeInvalidate` in `StoreBlogData` and `deleteBlogData`) and at the latest every 5 minutes (`postIndexTTL`), which also covers a change made through another instance.
 - **The search.** BM25 over stemmed words, in memory (`Index.Search`). Title words count five times, keyword words twice, text words once. `stem` is a crude suffix stripper (share, shares, shared and sharing meet); it only has to treat both sides alike. Filler words are dropped. A blog passage scores 0.85 of what the same words would score in a note, so a note wins when they are close. Only hits with at least half the best score are kept.
-- **Coverage.** Each hit also carries the share of the question's words, weighted by their rarity, that the passage contains. A word that no passage has counts in full: it means the question is about something else. This is what keeps platform text out of ordinary coding questions ("why does my loop never end" matches the word *loop* in one note and has 25% coverage), where a plain score would not.
+- **Coverage.** Each hit also carries the share of the question's words, weighted by their rarity, that the passage contains. A word that no passage has counts for 0.4 of its weight (`unknownWeight`): it says the question is about something else, but people also pad questions ("I mean ..."), so one such word must not decide it. This is what keeps platform text out of ordinary coding questions ("why does my loop never end" matches the word *loop* in one note and has 25% coverage), where a plain score would not.
 - **Two settings** (constants in `knowledge_proxy.go`). The Genie panel (`"context": "chat"`) needs a score of 2.0 and a coverage of 0.65 and takes at most 3 passages: it adds a note only when one clearly matches. The blog editor (`"context": "blog"`) needs a score of 1.5, no coverage, and takes at most 4: it wants the best ones. The thresholds were set with the example questions in `knowledge_test.go`; change them with those tests in view.
-- **The request.** Two body fields, never sent on to the model (`sanitizeChatBody` builds a fresh body): `context` (`chat` or `blog`; anything else, or none, is ignored, which is what the practice page and New question send) and `context_hint`. A hint, when not blank, replaces the last user message as the text to search with: the blog editor sends the post title and the text it works on, because its prompt is mostly instructions that would match the wrong notes. The editor sends the field only for the tasks that write new text ("Continue writing", "Write about this").
+- **The request.** Two body fields, never sent on to the model (`sanitizeChatBody` builds a fresh body): `context` (`chat` or `blog`; anything else, or none, is ignored, which is what the practice page and New question send) and `context_hint`. The panel puts a speaker tag in front of what a user writes (`[user-k3J9x] `, so that people in a shared session can be told apart); `lastUserText` removes it, because its random id is a word no note has and used to push the coverage of every question under the threshold. A hint, when not blank, replaces the last user message as the text to search with: the blog editor sends the post title and the text it works on, because its prompt is mostly instructions that would match the wrong notes. The editor sends the field only for the tasks that write new text ("Continue writing", "Write about this").
 - **The system message.** `addKnowledge` puts one system message in front of the conversation: the facts, each marked `[note]` or `[blog post]` with its title, at most 1,400 characters per passage and 5,000 in all, and the rule: use them to answer questions about OpenREPL, do not invent features, limits, prices or steps they do not mention, say you are not sure if they do not cover the question, use a blog post only if it answers the question and say it comes from a post. When nothing matched, the body is sent on unchanged.
 - **The answer.** The response carries `X-OpenREPL-Context` (also listed in `Access-Control-Expose-Headers`): `none` when the request asked and nothing matched, else base64url of `[{"id","title","kind"}]`. Without the field the header is absent. The Genie panel shows a line under each reply: "Answered without OpenREPL notes", or "Used OpenREPL notes:" with the titles as buttons (`appendContextLine`). A title opens a card in the panel with one row per passage; a row loads its text from `GET /knowledge/<id>` the first time it is opened and shows it as plain text, with "Read more in the docs" (or "Read the post") when it has a link, opened in a new tab and only if it is a path on this site. One row is open at a time; Escape closes the card (and not the panel), as does sending the next message. The blog editor shows no line.
 - **`GET /knowledge/<id>`.** Public, `Cache-Control: no-store`, GET and HEAD only. It returns `{id, kind, title, text, link}` for an id that the index holds, and 404 for everything else: the id must match `[a-z0-9-]{1,80}` and be a key of the index, so a path, `..`, an encoded slash or a file name is never looked at on disk. Blog passage ids are `post-<10 hex of sha1(name)>-<n>`.

src/resources/chat-widget/src/index.ts

@@ -181,7 +181,11 @@ function modelFields(): Record<string, any> {
 // What Genie is told about the page, and how many messages it keeps. An admin
 // can change the numbers (settings.js: site_settings.genieContext); these are
 // the built-in ones.
-const NUM_MANDATORY_ENTRIES = 4;
+// The messages at the start of the history that are never trimmed: what Genie
+// is told about itself, and the last of them is the editor snapshot that every
+// user message refreshes. Set in init() once they are in; it was a fixed 4
+// while two of them were the page text and the documentation list.
+let NUM_MANDATORY_ENTRIES = 4;
 function genieLimit(name: "editorChars" | "terminalChars" | "terminalLines" | "history", fallback: number): number {
   const n = Number((((window as any).site_settings || {}).genieContext || {})[name]);
   return n > 0 ? n : fallback;
@@ -359,14 +363,15 @@ async function init() {
   }
 
   let welcomeprompt = "welcome to openrepl.com!! you are Genie. An OpenRepl AI";
-  // only four permanent prompts
+  // the permanent prompts: who Genie is, (practice: the interviewer prompt), and the editor snapshot
   if (window.location.pathname.includes("practice")) {
     addMessageToHistory("system", welcomeprompt+" Interviewer.");
     addMessageToHistory("system", interviewPrompt);
   } else {
     addMessageToHistory("system", welcomeprompt+" Assistant.");
   }
   addMessageToHistory("system", "Openrepl IDE/EditorCodeContent: "+ fetchEditorContent());
+  NUM_MANDATORY_ENTRIES = conversationHistory.length; // the editor snapshot is the last of them
   setupFBListener();
 }
 window.addEventListener("load", init);

src/server/knowledge.go

@@ -70,7 +70,8 @@ func init() {
 		into is it its itself just me more most my myself no nor not now of off on once only or other our ours out over own same she should so some such
 		than that the their theirs them then there these they this those through to too under until up very was we were what when where which while who
 		whom why will with would you your yours yourself also get got make made need want like please tell show use used using way
-		someone anyone everyone somebody anybody everybody happen happens happened`) {
+		someone anyone everyone somebody anybody everybody happen happens happened
+		mean means meant know thing things sure ok okay yes yeah hi hello hey thanks thank`) {
 		stopwords[w] = true
 	}
 }
@@ -203,6 +204,10 @@ const (
 	postDiscount = 0.85
 	// a hit must score at least this share of the best one
 	relativeCut = 0.5
+	// a word that no passage has counts for this share of its weight in the
+	// coverage: it says the question wanders off the site, but people also pad
+	// questions with words of their own ("I mean ...")
+	unknownWeight = 0.4
 )
 
 // Search returns up to limit passages whose score reaches min and whose
@@ -227,7 +232,11 @@ func (ix *Index) Search(query string, limit int, min, minCoverage float64) []Hit
 	idfs := make([]float64, len(q))
 	for i, t := range q {
 		idfs[i] = logIDF(n, float64(ix.df[t]))
-		all += idfs[i]
+		if ix.df[t] == 0 {
+			all += idfs[i] * unknownWeight
+		} else {
+			all += idfs[i]
+		}
 	}
 	var hits []Hit
 	for _, p := range ix.passages {

src/server/knowledge_proxy.go

@@ -5,6 +5,7 @@ import (
 	"encoding/json"
 	"log"
 	"net/http"
+	"regexp"
 	"strings"
 	"sync"
 	"unicode/utf8"
@@ -130,8 +131,16 @@ func (c contextResult) header() string {
 	return base64.RawURLEncoding.EncodeToString(data)
 }
 
-// lastUserText is the text of the last message the user wrote.
+// speakerTag is what the chat panel puts in front of a message so that several
+// people in one shared session can be told apart: "[user-k3J9x] ".
+var speakerTag = regexp.MustCompile(`^\s*\[(?:user|system)-[A-Za-z0-9_-]{1,32}\]\s*`)
+
+// lastUserText is the text of the last message the user wrote, without the tag.
 func lastUserText(messages []json.RawMessage) string {
+	return speakerTag.ReplaceAllString(lastUserMessage(messages), "")
+}
+
+func lastUserMessage(messages []json.RawMessage) string {
 	for i := len(messages) - 1; i >= 0; i-- {
 		var m struct {
 			Role    string          `json:"role"`

src/server/knowledge_test.go

@@ -504,3 +504,122 @@ func TestAnUnopenedBlogStoreGivesNoPostsInsteadOfPanicking(t *testing.T) {
 		t.Error("the notes are missing without a blog store")
 	}
 }
+
+// The chat panel puts a tag in front of what a user writes ("[user-k3J9x] "), so
+// that people in a shared session can be told apart. The tag is not part of
+// the question: its random id is a word no note has, and used to push the
+// coverage of every question under the threshold.
+func TestTheSpeakerTagIsNotPartOfTheQuestion(t *testing.T) {
+	kb := testKB(t, nil)
+	for _, q := range []string{
+		"how do i use genie",
+		"how to fork",
+		"i mean how to fork the openrepl terminal",
+		"how can i share my terminal",
+		"give me the cling documentation",
+		"how do I upload a file",
+	} {
+		msgs := []map[string]string{
+			{"role": "system", "content": "you are Genie"},
+			{"role": "user", "content": "[user-Xk3j9Qa2] " + q},
+		}
+		_, res := addKnowledge(chatBody(t, map[string]interface{}{"context": "chat", "messages": msgs}), kb)
+		if len(res.Hits) == 0 {
+			t.Errorf("%q: nothing matched with the tag in front", q)
+		}
+	}
+	if got := lastUserText([]json.RawMessage{json.RawMessage(`{"role":"user","content":"[user-ab_c-1]   share it"}`)}); got != "share it" {
+		t.Errorf("tag not removed: %q", got)
+	}
+	if got := lastUserText([]json.RawMessage{json.RawMessage(`{"role":"user","content":"a [user-x] b"}`)}); got != "a [user-x] b" {
+		t.Errorf("a tag in the middle must stay: %q", got)
+	}
+}
+
+func TestPaddingWordsDoNotHideAQuestionAboutTheSite(t *testing.T) {
+	ix := testKB(t, nil).Index()
+	for q, want := range map[string]string{
+		"i mean how to fork the openrepl terminal": "fork-a-repl",
+		"hey, how do I share my session?":          "sharing-a-session",
+		"hey, what can genie do for me":            "genie",
+	} {
+		hits := ix.Search(q, chatContextLimit, chatContextMin, chatContextCoverage)
+		if len(hits) == 0 || hits[0].P.ID != want {
+			t.Errorf("%q -> %v, want %s", q, hits, want)
+		}
+	}
+	for _, q := range []string{"thanks that worked", "ok", "how do I sort a list", "what is my mean value in python"} {
+		if hits := ix.Search(q, chatContextLimit, chatContextMin, chatContextCoverage); len(hits) != 0 {
+			t.Errorf("%q matched %s", q, hits[0].P.ID)
+		}
+	}
+}
+
+// The request the chat panel really sends: its fixed messages, the tagged
+// question in the middle of the history, and the editor snapshot, a system
+// message, last. Every question of the bug report goes through the proxy.
+func TestTheRealWidgetRequestGetsItsNotesThroughTheProxy(t *testing.T) {
+	isolateSettings(t)
+	t.Setenv("OPENREPL_OPENAI_API_KEY", "sk-test-openai")
+	setKnowledge(testKB(t, nil))
+	t.Cleanup(func() { setKnowledge(nil) })
+	url, _, lastBody := fakeUpstream(t, 200, `{"choices":[{"message":{"content":"ok"}}]}`)
+	old := openaiEndpoint
+	openaiEndpoint = url
+	t.Cleanup(func() { openaiEndpoint = old })
+
+	widget := func(question string, earlier ...map[string]string) string {
+		msgs := []map[string]string{
+			{"role": "system", "content": "welcome to openrepl.com!! you are Genie. An OpenRepl AI Assistant."},
+			{"role": "system", "content": "Openrepl IDE real-time context of what the user is working on.\nLanguage: C and C++\n--- Editor code ---\nint main(){}\n--- Terminal output ---\ncling"},
+		}
+		msgs = append(msgs, earlier...)
+		msgs = append(msgs,
+			map[string]string{"role": "user", "content": "[user-a2592] " + question},
+			map[string]string{"role": "system", "content": "Openrepl IDE real-time context of what the user is working on.\nLanguage: C and C++\n--- Editor code ---\nint main(){}\n--- Terminal output ---\ncling"})
+		return string(chatBody(t, map[string]interface{}{
+			"model": "gpt-6-luna", "reasoning_effort": "low", "max_completion_tokens": 2000,
+			"messages": msgs, "stream": false, "context": "chat",
+		}))
+	}
+	earlier := []map[string]string{
+		{"role": "user", "content": "[user-a2592] hello"},
+		{"role": "assistant", "content": "Hi, how can I help?"},
+	}
+	for question, want := range map[string]string{
+		"how do i use genie": "genie",
+		"how to fork":        "fork-a-repl",
+		"i mean how to fork the openrepl terminal": "fork-a-repl",
+		"share":                           "sharing-a-session",
+		"how can i share my terminal":     "sharing-a-session",
+		"give me the cling documentation": "languages",
+	} {
+		for _, history := range [][]map[string]string{nil, earlier} {
+			w := proxyCall(t, widget(question, history...))
+			if w.Code != 200 {
+				t.Fatalf("%q: %d %s", question, w.Code, w.Body.String())
+			}
+			header := w.Header().Get(contextHeader)
+			if header == "" || header == "none" {
+				t.Errorf("%q (history %d): header %q, the notes were not found", question, len(history), header)
+				continue
+			}
+			ids := decodeContextHeader(t, header)
+			found := false
+			for _, n := range ids {
+				found = found || n["id"] == want
+			}
+			if !found {
+				t.Errorf("%q: used %v, want %s among them", question, ids, want)
+			}
+			if !strings.Contains(*lastBody, "Facts about OpenREPL") || strings.Contains(*lastBody, `"context"`) {
+				t.Errorf("%q: the model's request: %s", question, *lastBody)
+			}
+		}
+	}
+	// and a coding question still gets nothing
+	w := proxyCall(t, widget("why does my while loop never end"))
+	if w.Header().Get(contextHeader) != "none" || strings.Contains(*lastBody, "Facts about OpenREPL") {
+		t.Errorf("a coding question: header %q", w.Header().Get(contextHeader))
+	}
+}