Downloads archived PDF files from the Wayback Machine using a JSON list of source URLs.
- Reads URLs from JSON
- Canonicalizes
?PHPSESSIDvariants - Queries Wayback CDX for snapshots
- Tries newest snapshot first, then older ones
- Verifies downloaded files are real PDFs
- Skips already-downloaded valid files
- Preserves URL folder structure locally
- Python 3.10+
- requests
python crawler.py
Place a JSON file such as test.json next to the script.
- downloaded PDFs under configured destination
candidate_manifest.jsondownload_results.json
- Wayback can be slow
- URLs with
?PHPSESSIDmay exist as separate archived originals - multithreading can improve throughput but may increase rate limiting