Skip to content

About

Allows to crawl through Wayback and download pdfs

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Wayback PDF Crawler

Downloads archived PDF files from the Wayback Machine using a JSON list of source URLs.

Features

  • Reads URLs from JSON
  • Canonicalizes ?PHPSESSID variants
  • Queries Wayback CDX for snapshots
  • Tries newest snapshot first, then older ones
  • Verifies downloaded files are real PDFs
  • Skips already-downloaded valid files
  • Preserves URL folder structure locally

Requirements

  • Python 3.10+
  • requests

Usage

python crawler.py

Input

Place a JSON file such as test.json next to the script.

Output

  • downloaded PDFs under configured destination
  • candidate_manifest.json
  • download_results.json

Notes

  • Wayback can be slow
  • URLs with ?PHPSESSID may exist as separate archived originals
  • multithreading can improve throughput but may increase rate limiting

About

Allows to crawl through Wayback and download pdfs

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages