Refactor: Migrate data storage to PostgreSQL

[frenzel-ai-dev]

This commit introduces a significant refactoring to replace the flat-file based data storage and caching mechanism with a PostgreSQL database. This change addresses the issue of flat files becoming unwieldy for larger numbers of jobs and more complex querying, providing a more robust and scalable backend.

Key changes include:

1. Database Schema:

   • Defined new SQLAlchemy models (CachedSearches, JobDetails, JobAnalysisResults) in job_recommender/db_schema.py for storing cached search results, detailed job information, and (optionally) analysis results.

2. Database Utilities:

   • Added job_recommender/db_utils.py with functions to create a SQLAlchemy engine (create_db_engine) from environment variables and to create database tables (create_tables).

3. Cache Refactoring:

   • JobCache in job_recommender/cache.py now uses the cached_searches PostgreSQL table instead of JSON files for caching job scraping results.

4. Job Data Storage:

   • BaseJobScraper.save_jobs in job_recommender/job_scraper.py now saves job descriptions and details to the job_details PostgreSQL table, replacing the previous behavior of saving to individual text files.
   • ParallelJobScraper was updated to support this new saving mechanism.

5. Job Analysis Update:

   • JobAnalyzer.read_job_descriptions in job_recommender/job_analyzer.py now fetches job descriptions from the job_details table instead of reading from local text files. The CLI was updated to reflect this change.

6. Configuration & Documentation:

   • requirements.txt updated with SQLAlchemy and psycopg2-binary.
   • example.env updated with PostgreSQL connection variables.
   • README.md updated with database setup instructions, new dependencies, and revised usage instructions.

7. Data Migration:

   • Added migrate_data_to_db.py, a script to perform a one-time migration of data from old flat files (.cache/ and job_descriptions/) to the new PostgreSQL database.

8. Unit Tests:

   • Updated existing unit tests and added new ones for JobCache, BaseJobScraper, JobAnalyzer, and db_utils to use mocking for database interactions, ensuring coverage for the new PostgreSQL integration.

This migration enhances data management capabilities, improves query performance for cached data, and provides a more structured approach to storing job and analysis information.

[Copilot]

Pull Request Overview

This PR refactors job data storage from flat files to a PostgreSQL database, updating the underlying schema, caching and saving mechanisms as well as corresponding unit and integration tests.

• Introduces new SQLAlchemy models and utilities for database connection and table creation.
• Updates job scraping, caching, and analysis functionality to interact with PostgreSQL instead of the file system.
• Adjusts unit tests and CLI usage instructions to align with the new database-backed architecture.

Reviewed Changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/test_job_scraper.py	Extended tests to support DB engine injection and mocking.
tests/test_job_analyzer.py	Updated tests to use database interactions and fixed minor import issues.
tests/test_db_utils.py	Added tests for DB engine creation and table creation functions.
tests/test_cache.py	Modified cache tests to validate DB-backed caching logic.
migrate_data_to_db.py	Added data migration script from flat files to PostgreSQL.
job_recommender/parallel_scraper.py	Updated parallel scraper to use the DB-backed JobCache.
job_recommender/job_scraper.py	Refactored save_jobs to store job details in the database.
job_recommender/job_analyzer.py	Adjusted job description reading to use DB queries.
job_recommender/db_utils.py	New utility functions for creating a SQLAlchemy engine.
job_recommender/db_schema.py	New database schema for caching and job details.
job_recommender/cache.py	Updated caching to store and retrieve cache entries from PostgreSQL.
example.env	Updated environment file with database configuration.
README.md	Updated documentation for database setup and usage.

Comments suppressed due to low confidence (1)

tests/test_job_analyzer.py:5

• It appears there is a typo in the import statement; it should be 'from datetime import datetime' with a space between 'import' and 'datetime'.

from datetime importdatetime