Consolidate workflow patterns with bounded correction and recovery

.github/dependabot_test.py

@@ -0,0 +1,49 @@
+"""
+Utility helpers used by tests to verify Dependabot config wiring.
+The goal is to keep this lightweight and side-effect free.
+"""
+
+import json
+import os
+from pathlib import Path
+from typing import Dict, Optional
+
+
+def load_config(path: Optional[str] = None) -> Dict:
+    """Load dependabot configuration if it exists."""
+    target = Path(path or ".github/dependabot.yml")
+    if not target.exists():
+        return {}
+
+    try:
+        import yaml  # type: ignore
+    except Exception:
+        return {}
+
+    try:
+        return yaml.safe_load(target.read_text()) or {}
+    except Exception:
+        return {}
+
+
+def summarize_config(config: Optional[Dict] = None) -> Dict:
+    """Provide a minimal summary for assertions."""
+    cfg = config or load_config()
+    if not cfg:
+        return {"alerts": 0, "updates": []}
+
+    updates = cfg.get("updates", [])
+    return {
+        "alerts": len(updates),
+        "updates": [u.get("package-ecosystem") for u in updates],
+    }
+
+
+def main():
+    config = load_config()
+    summary = summarize_config(config)
+    print(json.dumps(summary))
+
+
+if __name__ == "__main__":
+    main()

.github/pull_request_template.md

@@ -1,27 +1,28 @@
-# Pull Request
+## Title
+<!-- Keep it concise; include the core change -->
 
-## Description
-Please include a summary of the change and which issue is fixed. Also include relevant motivation and context.
+## Summary
+- What changed:
+- Why it helps:
+- Orchestration example (optional):
+  ```python
+  # e.g., engine.execute_intent("...")
+  ```
 
-Fixes # (issue)
-
-## Type of change
+## Type
 - [ ] Bug fix
-- [ ] New feature
+- [ ] Feature
 - [ ] Breaking change
-- [ ] Documentation update
-- [ ] Other (describe):
+- [ ] Documentation
+- [ ] Other:
 
 ## Checklist
-- [ ] My code follows the style guidelines of this project
-- [ ] I have performed a self-review of my code
-- [ ] I have commented my code, particularly in hard-to-understand areas
-- [ ] I have made corresponding changes to the documentation
-- [ ] My changes generate no new warnings
-- [ ] I have added tests that prove my fix is effective or that my feature works
-- [ ] New and existing unit tests pass locally with my changes
-- [ ] Any dependent changes have been merged and published in downstream modules
-
-## Screenshots (if applicable)
+- [ ] Tests added/updated
+- [ ] Tests passing locally
+- [ ] Docs updated (if applicable)
+- [ ] Follows project style
 
-## Additional context 
+## Sections
+- Changes:
+- Testing:
+- Notes:

.github/workflows/python-ci.yml

@@ -5,9 +5,21 @@ on:
   push:
     paths:
       - '**.py'
+      - 'requirements-ci.txt'
+      - 'pytest.ini'
+      - 'workflows/data/**'
+      - 'workflows/*.ts'
+      - 'tools/typescript-validation/**'
+      - '.github/workflows/python-ci.yml'
   pull_request:
     paths:
       - '**.py'
+      - 'requirements-ci.txt'
+      - 'pytest.ini'
+      - 'workflows/data/**'
+      - 'workflows/*.ts'
+      - 'tools/typescript-validation/**'
+      - '.github/workflows/python-ci.yml'
 
 jobs:
   lint-test:
@@ -21,6 +33,11 @@ jobs:
         uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "22"
+      - name: Install TypeScript validator
+        run: npm ci --ignore-scripts --prefix tools/typescript-validation
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip

.gitignore

@@ -427,3 +427,6 @@ security_config.yaml.backup
 
 # ARC agent prototype generated artifacts
 agent_trajectory.png
+
+# Reviewed provenance for imported workflow definitions (not runtime data).
+!/workflows/source_manifest.json

MCP_INTEGRATION_PLAN.md

@@ -2,6 +2,7 @@
 
 ## Overview
 This plan systematically replaces all 60+ mock implementations with real, MCP-compliant code while maintaining the MCP-first architecture principles.
+No mock endpoints are permitted in production; every step below documents how we remove or replace mocks with real MCP connections.
 
 ## Phase 1: Core MCP Infrastructure (Immediate)
 

README.md

@@ -8,6 +8,7 @@
 - ✅ **Makefile** – Developer experience shortcuts
 - ✅ **README.md** – Comprehensive documentation
 - ✅ **mcp/main.py** – FastAPI MCP server implementation
+- ✅ **Workflow catalog** – `workflows/templates.py` consolidates WRKFLW + workflows patterns with self-correction
 
 ### Supporting infrastructure
 
@@ -27,12 +28,20 @@ make health
 
 Open http://localhost:8080/health to verify the API is running.
 
+WRKFLW and workflows repositories are archived. Their BuildPlan/agent/DAG, source intake, construction lifecycle, and artifact patterns are mapped into the canonical engine alongside the original templates. See [source coverage and runnable examples](docs/workflows/IMPORTED_PATTERNS.md) for the exact contracts and required host services.
+You can extend the catalog by dropping yaml/json workflow definitions into `workflows/data/`; they are auto-loaded when the catalog module is imported.
+
 ## Development
 
-The workspace includes a **DevContainer** definition – simply open the folder in VS Code or Cursor and choose *Reopen in Container* to get an IDE connected to the running services. 
+The workspace includes a **DevContainer** definition – simply open the folder in VS Code or Cursor and choose *Reopen in Container* to get an IDE connected to the running services.
+
+## Workflow consolidation runtime status
+
+The catalog auto-loads external definitions, but loader success alone is not runtime readiness. The construction flow now accepts explicit project records and calls the real analyzer and source-grounded summary connector. The TypeScript pipeline generates and compiles actual handlers, checks a supplied service URL and caches schema/report metadata using explicit Redis settings. Live provider access and integration requirements are described in [workflow execution and limits](docs/workflows/CONSOLIDATION.md). The additional imported lifecycle adapters execute RFI state transitions and configured vector-retrieval contracts; live provider acceptance, web/mobile UI parity, and external deployment remain separate. See [imported patterns](docs/workflows/IMPORTED_PATTERNS.md).
 
 ## Repo assist
 
 For a repo-specific map of the current tree, dependency snapshot, mermaid
 architecture diagram, MCP capability-verification notes, and agent workflow
-checklist, see [`docs/REPO_MAP.md`](docs/REPO_MAP.md).
+checklist, see [`docs/REPO_MAP.md`](docs/REPO_MAP.md).
+

agents/specialized/code_generator.py

@@ -4,6 +4,7 @@
 import textwrap
 from datetime import datetime
 from typing import Dict, Any
+from workflows.service_scaffold import generate_typescript_service
 
 
 class CodeGeneratorAgent:
@@ -35,8 +36,8 @@ async def {function_name}({parameters}):
                             "result": result,
                             "timestamp": datetime.utcnow().isoformat()
                         }}
-                    except ValidationError as e:
-                        raise HTTPException(status_code=400, detail=str(e))
+                    except HTTPException:
+                        raise
                     except Exception as e:
                         raise HTTPException(status_code=500, detail=str(e))
             """),
@@ -91,7 +92,9 @@ async def execute(self, inputs: Dict[str, Any]) -> Dict[str, Any]:
         generation_type = self._parse_intent(intent)
 
         # Generate appropriate code
-        if generation_type == "api_endpoint":
+        if generation_type == "typescript_service":
+            code = generate_typescript_service(context)
+        elif generation_type == "api_endpoint":
             code = self._generate_api_endpoint(context)
         elif generation_type == "crud":
             code = self._generate_crud_api(context)
@@ -104,7 +107,11 @@ async def execute(self, inputs: Dict[str, Any]) -> Dict[str, Any]:
             "success": True,
             "generated_code": code,
             "generation_type": generation_type,
-            "files_created": self._get_file_list(code),
+            "files_created": (
+                [item["path"] for item in code["files"]]
+                if isinstance(code, dict)
+                else self._get_file_list(code)
+            ),
             "instructions": self._get_implementation_instructions(generation_type),
             "timestamp": datetime.utcnow().isoformat(),
         }
@@ -113,7 +120,9 @@ def _parse_intent(self, intent: str) -> str:
         """Parse intent to determine generation type"""
         intent_lower = intent.lower()
 
-        if "crud" in intent_lower:
+        if "typescript" in intent_lower:
+            return "typescript_service"
+        elif "crud" in intent_lower:
             return "crud"
         elif "model" in intent_lower or "schema" in intent_lower:
             return "data_model"
@@ -144,17 +153,21 @@ def _generate_api_endpoint(self, context: Dict) -> str:
             for param in params:
                 validation_logic += (
                     f"if not {param}:\n            "
-                    f"raise ValidationError('{param} is required')\n        "
+                    f"raise HTTPException(status_code=400, detail='{param} is required')\n        "
                 )
         else:
             validation_logic += "pass"
 
-        # Generate processing logic
-        processing_logic = """# Main processing logic
-        result = {
-            'processed': True,
-            'data': request
-        }"""
+        # Scaffolding returns the declared arguments; never reference a nonexistent request.
+        input_expression = (
+            "{" + ", ".join(f"{name!r}: {name}" for name in params) + "}"
+            if params
+            else "request"
+        )
+        processing_logic = (
+            "# Add application-specific behavior before deployment\n        "
+            f"result = {{'processed': True, 'data': {input_expression}}}"
+        )
 
         return self.templates["fastapi_endpoint"].format(
             endpoint_name=endpoint_name,
@@ -231,6 +244,7 @@ def _get_file_list(self, code: str) -> list:
     def _get_implementation_instructions(self, generation_type: str) -> str:
         """Get instructions for implementing generated code"""
         instructions = {
+            "typescript_service": "Write the artifact files, run npm install and npm run build, then supply a Store implementation and HTTP adapter. See generated README.md; no deployment was performed.",
             "api_endpoint": "Add this endpoint to your main FastAPI application (mcp/main.py)",
             "crud": "Create a new file 'crud_api.py' and import it in your main application",
             "data_model": "Add this model to your models.py file or create one if it doesn't exist",

analyzers/pattern_detector.py

@@ -6,6 +6,8 @@
 import asyncio
 import logging
 from datetime import datetime, timedelta
+from copy import deepcopy
+from collections import Counter
 from typing import Any, Dict, List
 
 from utils.db_tracker import ensure_tables_exist, get_db_connection
@@ -257,15 +259,72 @@ async def generate_mutation_recommendations(
 
         return recommendations
 
-    async def analyze(self) -> Dict[str, Any]:
+    def analyze_supplied_data(self, data):
+        """Aggregate observed supplied records, without inferred risk thresholds."""
+        if not isinstance(data, dict) or data.get("success") is not True:
+            raise ValueError("Analyzer requires a successful data-processing result")
+        if data.get("kind") == "field_capture":
+            from workflows.field_capture import normalize_field_capture
+
+            # Revalidate rather than trusting a caller-supplied provenance envelope.
+            normalized = normalize_field_capture(data["capture"])
+            records = normalized["records"]
+            patterns = {
+                "record_counts": {name: len(items) for name, items in records.items()},
+                "status_counts": {
+                    name: dict(
+                        Counter(
+                            item["status"]
+                            for item in items
+                            if isinstance(item.get("status"), str)
+                        )
+                    )
+                    for name, items in records.items()
+                },
+            }
+            insights = [
+                {
+                    "type": "field_capture",
+                    "project_id": normalized["project_id"],
+                    "report_date": normalized["report_date"],
+                    "sources": normalized["sources"],
+                    "observations": deepcopy(patterns),
+                }
+            ]
+        elif all(
+            key in data for key in ("files_processed", "total_records", "insights")
+        ):
+            patterns = {
+                "file_profile": {
+                    "directory": data.get("directory"),
+                    "files_processed": data["files_processed"],
+                    "total_records": data["total_records"],
+                }
+            }
+            insights = deepcopy(data["insights"])
+        else:
+            raise ValueError("Unsupported normalized data for pattern_detector")
+        return {
+            "success": True,
+            "patterns": patterns,
+            "insights": insights,
+            "recommendations": [],
+            "timestamp": datetime.utcnow().isoformat(),
+        }
+
+    async def analyze(self, data=None) -> Dict[str, Any]:
         """
         Analyze execution history and generate insights and recommendations.
 
         Returns:
             Dict[str, Any]: Analysis results
         """
         try:
+            if data is not None:
+                return self.analyze_supplied_data(data)
             patterns = await self.detect_patterns()
+            if patterns.get("error"):
+                raise RuntimeError(patterns["error"])
 
             insights = await self.generate_insights(
                 patterns["failure_patterns"],

config/__init__.py

@@ -0,0 +1 @@
+"""Configuration package for MCP and workflow execution."""