Goal: Scrape the Star Trek Wiki pages listing canon planets to determine what percentage of planets are habitable, and similarly scraped the personnel list to find the distribution of named officers by rank.
Quick note: all results are biased based on what the wiki editors have made a record of. Not all Starfleet officer's pages, for example, have their rank listed.
Technical components: Python script that uses Beautiful Soup (chosen for its simplicity, we do not have to scrape any dynamically generated pages). Analyze and visualize results in a Jupyter Notebook using Pandas and MatPlotLib.
- Setup environments
- Pick a webscraping utility
- Understand the format of planet status on wiki
- Write scraping script
- Test on subset of planets and manually confirm
- Run on all planets
- Data Visualization
- Repeat for Starfleet ranks
Contributors: Isaac Exposito, Liza George