feat: optional RAG layer and documentation updates

Author: Pro0f

CONTRIBUTING.md

@@ -32,20 +32,58 @@ pytest tests/test_integration.py
 
 ## Adding a New Adapter
 
-1. Create `src/devscontext/adapters/your_source.py`:
-   - Extend the `Adapter` base class
-   - Implement `fetch_context(task_id)` and `health_check()`
-   - Add any source-specific methods (e.g., `search()`)
+1. **Create adapter file** `src/devscontext/adapters/your_source.py`:
+   - Extend the `Adapter` base class from `plugins/base.py`
+   - Set class attributes: `name`, `source_type`, `config_schema`
+   - Implement required methods:
+     - `fetch_task_context(task_id, ticket=None) -> SourceContext`
+     - `search(query, max_results) -> list[SearchResult]`
+     - `health_check() -> bool`
+   - Optionally implement `close()` and `format_for_synthesis()`
 
-2. Add config model to `src/devscontext/config.py`
+2. **Add config model** to `src/devscontext/models.py`:
+   - Create Pydantic model with `enabled`, `primary` fields
+   - Add to `SourcesConfig`
 
-3. Wire into `src/devscontext/core.py`:
-   - Add adapter initialization
-   - Include in fetch/search flows
+3. **Register in plugin registry** `src/devscontext/plugins/registry.py`:
+   - Add to `register_builtin_plugins()` method
+   - Add loading logic in `load_from_config()`
 
-4. Add tests in `tests/test_your_source.py`
+4. **Add tests** in `tests/test_your_source.py`
 
-See `adapters/jira.py` as reference.
+See `adapters/slack.py` as a complete reference for communication sources.
+
+For full details, see [docs/plugins.md](docs/plugins.md).
+
+## Plugin Development
+
+DevsContext uses a plugin system for extensibility:
+
+- **Adapters**: Fetch context from data sources
+- **Synthesis Plugins**: Process and combine context
+
+External plugins can be published as pip packages using entry points.
+See [docs/plugins.md](docs/plugins.md) for the complete guide.
+
+## Testing Adapters
+
+```bash
+# Unit tests (no API keys needed)
+pytest tests/test_your_source.py
+
+# Test with real API (integration)
+devscontext test --task YOUR-123
+```
+
+## Testing Pre-processing Agent
+
+```bash
+# Process a single ticket
+devscontext agent process TEST-123
+
+# Check storage status
+devscontext agent status
+```
 
 ## Code Quality
 

README.md

@@ -80,42 +80,81 @@ Then in Claude Code:
 
 ## Supported Sources
 
-| Source | What's Fetched |
-|--------|----------------|
-| **Jira** | Ticket details, comments, linked issues, acceptance criteria |
-| **Fireflies** | Meeting transcripts, decisions, action items |
-| **Local Docs** | Architecture docs, coding standards, ADRs |
+| Source | What's Fetched | Status |
+|--------|----------------|--------|
+| **Jira** | Ticket details, comments, linked issues, acceptance criteria | Stable |
+| **Fireflies** | Meeting transcripts, decisions, action items | Stable |
+| **Local Docs** | Architecture docs, coding standards, ADRs | Stable |
+| **Slack** | Channel discussions, threads, decisions | New |
+| **Gmail** | Email threads related to tickets | New |
 
-Coming soon: Linear, Notion, Confluence, Slack threads
+Coming soon: Linear, Notion, Confluence
+
+## Pre-processing Agent
+
+Build context proactively before developers pick up tickets:
+
+```bash
+# Start the agent (polls Jira for ready tickets)
+devscontext agent start
+
+# Single run for CI/cron
+devscontext agent run-once
+
+# Check pre-built context status
+devscontext agent status
+```
+
+Configure in `.devscontext.yaml`:
+
+```yaml
+agents:
+  preprocessor:
+    enabled: true
+    jira_status: "Ready for Development"
+    jira_project: "PROJ"
+```
+
+See [docs/pre-processing.md](docs/pre-processing.md) for the full guide.
+
+## Plugin System
+
+DevsContext uses a plugin architecture for adapters and synthesis:
+
+- **Adapters**: Fetch context from sources (Jira, Slack, docs, etc.)
+- **Synthesis Plugins**: Combine context (LLM, template, passthrough)
+
+See [docs/plugins.md](docs/plugins.md) for creating custom plugins.
 
 ## Configuration
 
 DevsContext uses `.devscontext.yaml` in your project root:
 
 ```yaml
-adapters:
+sources:
   jira:
     enabled: true
     base_url: "https://your-company.atlassian.net"
     email: "${JIRA_EMAIL}"
     api_token: "${JIRA_API_TOKEN}"
 
-  fireflies:
-    enabled: false  # Optional
-    api_key: "${FIREFLIES_API_KEY}"
-
-  local_docs:
+  docs:
     enabled: true
     paths:
       - "./docs"
       - "./CLAUDE.md"
 
+  slack:
+    enabled: true
+    bot_token: "${SLACK_BOT_TOKEN}"
+    channels: ["engineering", "payments-team"]
+
 synthesis:
   provider: "anthropic"
-  model: "claude-3-haiku-20240307"
+  model: "claude-haiku-4-5"
 ```
 
-See [.devscontext.yaml.example](.devscontext.yaml.example) for all options.
+Full configuration reference: [docs/configuration.md](docs/configuration.md)
 
 ## How It Works