Proposal: Persistent Chat History & Optional Semantic Caching

[Alok-Kumar2005]

Description

Currently, the Knowledge Space Agent stores chat history in memory (for example, self.chat_history). While this works for local development and single-session usage, the history is lost when the server restarts or when deployed in stateless environments such as container-based or serverless deployments.

Additionally, repeated queries always trigger fresh LLM calls, which may increase latency and operational cost.

This issue proposes exploring a more scalable and modular approach to handling chat history and caching.

---

Motivation

1. Chat history is not persistent across restarts.
2. Stateless deployments may lose session context.
3. Repeated or identical queries always invoke the LLM.
4. There is currently no caching layer to reduce redundant computation.

Improving this could enhance scalability, reduce cost, and improve reproducibility.

---

Proposed Solution (Seeking Maintainer Feedback)

1. Persistent Chat History

Introduce a modular storage layer for chat history:

• SQLite for local development
• PostgreSQL for production
• Session-based storage (user/session ID mapping)

The default behavior can remain unchanged (in-memory) unless explicitly enabled via configuration.

---

2. Optional Caching Layer

Introduce a lightweight caching mechanism:

• Exact-match caching (Redis or in-memory cache)
• Future extension: semantic caching using embeddings

Caching should remain configurable and optional to avoid introducing mandatory infrastructure dependencies.

---

Scope of Work

• Abstract chat history into a storage interface
• Implement optional database-backed storage
• Add optional caching layer
• Add configuration flags (e.g., ENABLE_PERSISTENCE, ENABLE_CACHE)
• Update documentation accordingly

---

Expected Outcome

• Improved scalability
• Reduced redundant LLM calls
• Better support for stateless deployments
• Optional and backward-compatible implementation

---

I am happy to adjust the scope based on maintainer feedback before starting implementation.

[tisha-varma]

Hi! 👋 I’d love to work on this issue.

I’ve started a local implementation that:

• Abstracts chat history into a ChatStorage interface
• Adds an optional SQLite-backed persistence layer (stdlib only, zero new dependencies)
• Provides an optional in-memory cache with TTL and LRU eviction
• Keeps backward compatibility so default behaviour remains in-memory
• Uses configuration flags (e.g., ENABLE_PERSISTENCE, ENABLE_CACHE) defined in .env.template
• Includes unit tests covering storage and caching layers

If this direction looks good, I’ll open a PR for review. Happy to adjust based on feedback!

Thanks!

[ulekarsiddhant0-boop]

Hi @Alok-Kumar2005 , I am interested to resolve #61 issue. So, please assign this issue to me. Thank you!