Improve chat agent prompts, citation linking, and tool-call handling

[noor-lpi]

Summary

This branch covers several rounds of chat-agent behavior fixes:

• Prompt iterations 1 & 2 (cd5cef0, 6c540c6): rewrote the agent system prompt, fixed language detection, added citation grounding, and fixed a multi-tool-call issue.
• One-shot tool call + docs panel fix (7bdcfc7, reverted c8a562c, reapplied 37ce28e): caps the retrieval tool to one call per turn, and stops follow-up turns without a tool call from clearing the frontend's docs panel.
• Round 2 (this session) — five independent, cherry-pickable commits fixing issues reported from real usage:
  • fix(chat): enforce response brevity as a hard constraint — responses were producing full course structures from minimal context (e.g. a user just stating their role).
  • fix(chat): forbid links outside retrieved documents — the agent was still emitting links from its own training data, often stale/dead URLs.
  • feat(chat): add Socratic clarifying-question behavior — the agent now asks up to 3 targeted clarifying questions at the start of a conversation or on a topic shift, instead of answering at length on minimal context.
  • fix(chat): reorient follow-up questions to WeLearn's pedagogical use case — both the agent's own follow-up question and the separate /reformulate/questions suggestions now focus on moving a professor from understanding a topic to applying it in their teaching, instead of generic "keep learning about SDGs" framing.
  • fix(chat): auto-link citations the model forgets to format — adds a regex-based fallback (linkify_missing_citations) that catches citation markers the model left unlinked and wraps them using the URL already returned by the retrieval tool, applied to both the streaming and non-streaming chat endpoints.
• Unrelated small fix: Python 3.12 f-string nesting compatibility issue in src/app/tutor/service/agents.py.

Two issues investigated but not addressed in this branch (documented, deferred):

• A "stop generating" button — no cancellation mechanism exists in the backend today; needs a new mechanism here plus client-side wiring in the separate client repo.
• Source links not clickable until streaming finishes — traced to the API already sending citation data early in the stream; looks like a client-side rendering issue, not an API bug.

Test plan

• flake8 / isort / black clean on all changed files
• New unit tests for linkify_missing_citations in src/app/tests/services/test_helpers.py
• Full make test-poetry run (blocked locally by an unrelated transformers/huggingface_hub version mismatch tied to an in-progress dependency bump — re-run once that's resolved)
• Manual check via make run-poetry: confirm short responses + clarifying questions on a minimal prompt, no external links, and citations render as clickable links

🤖 Generated with Claude Code