Concurrent Web Crawler

This project evaluates the performance of concurrent web crawler using three different approaches.

• Sequential
• Parallel
• Concurrent

This project is based on my bachelor's Hardware Architect Lecture. The task is to implement a web crawler using Multi-Threading, Multi-Processing, and Asynchronous Programming.

The goal is to visualise how different concurrency approaches handle I/O tasks and affect the performance of the crawler.

Project Design

We crawl a set of pages starting from a seed URL and measure how long each method takes to complete.

Concurrency Models

• Sequential: Process one URL at a time.
• Multi-Threading: Using multiple threads within a single process.
• Multi-Processing: Using multiple processes.
• Asynchronous: Using asynchronous programming.

To calculate the efficiency of our parallel versions, we use the Speedup Factor:$$\text{Speedup} = \frac{T_{\text{sequential}}}{T_{\text{parallel}}}$$

Setup

1. requirements:

• python 3.8
• requests
• beautifulsoup4
• tqdm
• viztracer

2. Evaluation:

To run the crawler and view the results:

python -m viztracer webCrawler.py

vizviewer result.json

Results: