AI Agent Information Retrieval Project

Project Overview

This AI agent is designed to automatically retrieve and extract specific information from web sources based on user-defined queries. Users can upload a CSV or connect a Google Sheet, specify search parameters, and obtain structured data about entities in their dataset.

Features

• CSV and Google Sheets file upload
• Dynamic web search for each entity
• Custom prompt-based information retrieval
• LLM-powered data extraction
• Results display and CSV download

Prerequisites

• Python 3.9+
• API Keys:
  • Search API (SerpAPI)
  • LLM API (Groq)
  • Google Sheets API

Installation

1. Clone the repository

git clone https://github.com/rohraharjas/llmscraper.git
cd ai-agent-project

2. Install dependencies

pip install -r requirements.txt

3. Set up environment variables

• Create a .env file with:

  SERP_API_KEY=your_api_key
  GROQ_API_KEY=your_llm_api_key
  GSHEETS_API_KEY=your_gsheet_api_key
  

Usage

1. Run the dashboard

streamlit run app.py

2. Upload your data file
3. Select the primary column
4. Enter your custom search query
5. Retrieve and download results

Optional Advanced Features

• Multiple field extraction
• Direct Google Sheets output
• Robust error handling

Technologies Used

• Dashboard: Streamlit
• Data Handling: pandas
• Search API: SerpAPI
• LLM: Groq
• Backend: Python
• Agents: Langchain

Limitations

• Respects web scraping terms of service
• Subject to API rate limits
• Dependent on web data availability

Contributing

Please read CONTRIBUTING.md for details on our code of conduct and the process for submitting pull requests.

License

This project is licensed under the MIT License - see the LICENSE.md file for details.