📚 Knowledge Document Creation System

A comprehensive AI-powered system for extracting, analyzing, and synthesizing knowledge from PDF documents using Ollama's local AI models.

🌟 Features

• 📄 PDF Processing: Extract text, tables, and images from PDF documents
• 🤖 AI-Powered Analysis: Uses Ollama with Llama 3.1 for intelligent content analysis
• 📊 Table Extraction: Automatically detects and extracts tables from PDFs
• 🎨 Flowchart Analysis: Analyzes flowcharts and diagrams using AI vision capabilities
• 🔗 Citation Mapping: Maps citations to reference documents
• 🔍 Similarity Search: Finds relevant content across documents using FAISS
• 📝 Knowledge Synthesis: Creates comprehensive knowledge documents
• 💻 Streamlit Interface: User-friendly web interface for document processing

🚀 Quick Start

Prerequisites

• Python 3.8 or higher
• Ollama installed locally
• Git

Installation

1. Clone the repository

   git clone <repository-url>
   cd knowledge-document-creation

2. Create and activate virtual environment

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies

   pip install -r requirements.txt

4. Download NLTK data

   python -c "import nltk; nltk.download('punkt_tab'); nltk.download('punkt'); nltk.download('averaged_perceptron_tagger'); nltk.download('stopwords')"

5. Setup Ollama

   # Install Ollama from https://ollama.ai/
   ollama serve
   ollama pull llama3.1:8b

6. Run the application

   streamlit run main.py

7. Open your browser to http://localhost:8501

📖 Usage

Basic Workflow

1. Upload Main PDF: Upload the primary document you want to analyze
2. Upload Reference PDFs: Upload reference documents for comparison and citation mapping
3. Generate Knowledge Document: Click "Generate" to process all documents
4. View Results: Review the generated knowledge document with statistics and organized content
5. Download: Download the document in Word or text format

Document Processing Pipeline

PDF Input → Text Extraction → Table Detection → Flowchart Analysis → AI Processing → Knowledge Synthesis → Output

🏗️ Project Structure

knowledge-document-creation/
├── main.py                          # Main Streamlit application
├── requirements.txt                 # Python dependencies
├── setup_environment.py            # Environment setup script
├── .gitignore                      # Git ignore file
├── README.md                       # This file
├── Paragraph_extraction_from_pdf.py # Text extraction module
├── Table_to_text/                  # Table extraction module
│   └── Detect_table_redact.py
├── Flowchart_to_text/              # Flowchart analysis module
│   ├── flowchart_to_text.py
│   └── Detect_flowchart_redact.py
└── csv/                           # Generated CSV files (ignored by git)

🔧 Configuration

Environment Variables

The application uses the following environment variables:

• LLM_PROVIDER=ollama
• OLLAMA_MODEL=llama3.1:8b

These are automatically set by the setup script.

Ollama Models

The system is configured to use llama3.1:8b by default. You can use other models by:

1. Pulling the desired model: ollama pull <model-name>
2. Setting the environment variable: export OLLAMA_MODEL=<model-name>

📊 Features in Detail

Text Extraction

• Extracts text from PDFs using multiple libraries (PyMuPDF, pdfplumber)
• Handles various PDF formats and layouts
• Preserves text structure and formatting

Table Detection

• Automatically identifies tables in PDF documents
• Extracts table content to Excel format
• Maps table references in the main document

Flowchart Analysis

• Extracts images and diagrams from PDFs
• Uses Ollama's vision capabilities to analyze flowcharts
• Generates textual descriptions of visual content

Citation Mapping

• Identifies citation patterns in academic documents
• Maps citations to corresponding reference documents
• Performs similarity search to find relevant content

Knowledge Synthesis

• Combines extracted content from all sources
• Creates organized knowledge documents
• Provides download options in multiple formats

🛠️ Development

Adding New Features

1. New Document Types: Extend the extraction modules in Table_to_text/ or Flowchart_to_text/
2. AI Models: Modify the Ollama integration in flowchart_to_text.py
3. UI Components: Add new Streamlit components in main.py

Testing

# Test imports
python -c "from main import *; print('✅ All imports successful')"

# Test specific modules
python -c "from Paragraph_extraction_from_pdf import create_csv; print('✅ Text extraction works')"

🐛 Troubleshooting

Common Issues

1. Ollama Connection Error

   # Check if Ollama is running
   ollama list
   # Start Ollama service
   ollama serve

2. NLTK Data Missing

   python -c "import nltk; nltk.download('punkt_tab')"

3. PyMuPDF Import Error

   pip uninstall PyMuPDF
   pip install pymupdf

4. Permission Errors

   • Ensure write permissions in the project directory
   • Check if antivirus software is blocking file operations

Getting Help

• Check the Issues page
• Review the troubleshooting section above
• Ensure all dependencies are properly installed

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

🙏 Acknowledgments

• Ollama for local AI model serving
• Streamlit for the web interface
• PyMuPDF for PDF processing
• LangChain for document processing
• FAISS for similarity search

📞 Support

For support, email your-email@example.com or create an issue in the repository.

---

Made with ❤️ for knowledge discovery and document analysis