added questions, objectives, key points for 3 episodes, and setup instructions

Author: josenino95

episodes/a-real-website.md

@@ -6,14 +6,16 @@ exercises: 15
 
 :::::::::::::::::::::::::::::::::::::: questions 
 
-- How do you write a lesson using Markdown and `{sandpaper}`?
+- How can I get the data and information from a real website?
+- How can I start automating my web scraping tasks?
 
 ::::::::::::::::::::::::::::::::::::::::::::::::
 
 ::::::::::::::::::::::::::::::::::::: objectives
 
-- Explain how to use markdown with The Carpentries Workbench
-- Demonstrate how to include pieces of code, figures, and nested challenge blocks
+- Use the `requests` package to get the HTML document behind a website.
+- Navigate the tree structure behind an HTML document to extract the information we need.
+- Know how to avoid being blocked by sending too much requests to a website.
 
 ::::::::::::::::::::::::::::::::::::::::::::::::
 
@@ -139,7 +141,7 @@ print(workshops_table.prettify())
 </table>
 ```
 
-To navigate in this HTML document tree we can use the methods `.contents()` (to access direct children nodes), `.parent()` (to access the parent node), `.next_sibling()`, and `.previous_sibling()` (to access the siblings of a node) methods. For example, if we want to access the second row of the table, which is the second child of the table element we could use the following code.
+To navigate in this HTML document tree we can use the following properties of the "bs4.element.Tag" object: `.contents` (to access direct children nodes), `.parent` (to access the parent node), `.next_sibling`, and `.previous_sibling` (to access the siblings of a node) methods. For example, if we want to access the second row of the table, which is the second child of the table element we could use the following code.
 
 ```python
 # The second [1 in Python indexing] child of our table element
@@ -424,10 +426,9 @@ print(req.status_code)
 
 ::::::::::::::::::::::::::::::::::::: keypoints 
 
-- Use `.md` files for episodes when you want static content
-- Use `.Rmd` files for episodes when you need to generate output
-- Run `sandpaper::check_lesson()` to identify any issues with your lesson
-- Run `sandpaper::build_lesson()` to preview your lesson locally
+- We can get the HTML behind any website using the "requests" package and the function `requests.get('website_url').text`.
+- An HTML document is a nested tree of elements. Therefore, from a given element, we can access its child, parent, or sibling, using `.contents`, `.parent`, `.next_sibling`, and `previous_sibling`.
+- It's polite to not send too many requests to a website in a short period of time. For that, we can use the `sleep()` function of the built-in Python module `time`. 
 
 ::::::::::::::::::::::::::::::::::::::::::::::::
 

episodes/dynamic-websites.md

@@ -6,14 +6,17 @@ exercises: 5
 
 :::::::::::::::::::::::::::::::::::::: questions 
 
-- How do you write a lesson using Markdown and `{sandpaper}`?
+- What are the differences between static and dynamic websites?
+- Why is it important to understand these differences when doing web scraping?
+- How can I start my own web scraping project?
 
 ::::::::::::::::::::::::::::::::::::::::::::::::
 
 ::::::::::::::::::::::::::::::::::::: objectives
 
-- Explain how to use markdown with The Carpentries Workbench
-- Demonstrate how to include pieces of code, figures, and nested challenge blocks
+- Use the `Selenium` package to scrape dynamic websites.
+- Identify the elements of interest using the browser's "Inspect" tool.
+- Understand the usual pipeline of a web scraping project.
 
 ::::::::::::::::::::::::::::::::::::::::::::::::
 
@@ -257,11 +260,17 @@ This scraping pipeline helps break down complex scraping tasks into manageable s
 
 ::::::::::::::::::::::::::::::::::::: keypoints 
 
-- Use `.md` files for episodes when you want static content
-- Use `.Rmd` files for episodes when you need to generate output
-- Run `sandpaper::check_lesson()` to identify any issues with your lesson
-- Run `sandpaper::build_lesson()` to preview your lesson locally
-
+- Dynamic websites load content using JavaScript, which isn't present in the initial or source HTML. It's important to distinguish between static and dynamic content when planning your scraping approach.
+- The `Selenium` package and its `webdriver` module simulate a real user interacting with a browser, allowing it to execute JavaScript and clicking, scrolling or filling in text boxes.
+- Here are the commandand we learned when we use `Selenium`:
+  - `webdriver.Chrome()` # Start the Google Chrome browser simulator
+  - `.get("website_url")` # Go to a given website
+  - `.find_element(by, value)` and `.find_elements(by, value)` # Get a given element
+  - `.click()` # Click the element selected
+  - `.page_source` # Get the HTML after JavaScript has executed, which can later be parsed with BeautifulSoup
+  - `.quit()` # Close the browser simulator
+- The browser's "Inspect" tool allows users to view the HTML document after dynamic content has loaded, revealing elements added by JavaScript. This tool helps identify the specific elements you are interested in scraping.
+- A typical scraping pipeline involves understanding the website's structure, determining content type (static or dynamic), using the appropriate tools (requests and BeautifulSoup for static, Selenium and BeautifulSoup for dynamic), and structuring the scraped data for analysis.
 ::::::::::::::::::::::::::::::::::::::::::::::::
 
 [r-markdown]: https://rmarkdown.rstudio.com/

episodes/hello-scraping.md

@@ -1,19 +1,21 @@
 ---
 title: "Hello-Scraping"
-teaching: 30
-exercises: 5
+teaching: 40
+exercises: 10
 ---
 
 :::::::::::::::::::::::::::::::::::::: questions 
 
-- How do you write a lesson using Markdown and `{sandpaper}`?
+- What is behind a website and how can I extract its information?
+- What is there to consider before I do web scraping?
 
 ::::::::::::::::::::::::::::::::::::::::::::::::
 
 ::::::::::::::::::::::::::::::::::::: objectives
 
-- Explain how to use markdown with The Carpentries Workbench
-- Demonstrate how to include pieces of code, figures, and nested challenge blocks
+- Identify the structure and basic components of an HTML document.
+- Use BeautifulSoup to locate elements, tags, attributes and text in an HTML document.
+- Understand the situations in which web scraping is not suitable for obtaining the desired data.
 
 ::::::::::::::::::::::::::::::::::::::::::::::::
 
@@ -320,10 +322,12 @@ To conclude, here is a brief code of conduct you should consider when doing web
 
 ::::::::::::::::::::::::::::::::::::: keypoints 
 
-- Use `.md` files for episodes when you want static content
-- Use `.Rmd` files for episodes when you need to generate output
-- Run `sandpaper::check_lesson()` to identify any issues with your lesson
-- Run `sandpaper::build_lesson()` to preview your lesson locally
+- Every website has an HTML document behind it that gives a structure to its content.
+- An HTML is composed of elements, which usually have a opening `<tag>` and a closing `</tag>`.
+- Elements can have different properties, assigned by attributes in the form of `<tag attribute_name="value">`.
+- We can parse any HTML document with `BeautifulSoup()` and find elements using the `.find()` and `.find_all()` methods.
+- We can access the text of an element using the `.get_text()` method and the attribute values as we do with Python dictionaries (`element["attribute_name"]`).
+- We must be careful to not tresspass the Terms of Service (TOS) of the website we are scraping.
 
 ::::::::::::::::::::::::::::::::::::::::::::::::
 

learners/setup.md

@@ -2,54 +2,21 @@
 title: Setup
 ---
 
-FIXME: 
-- Python, Jupyter notebook, libraries: b4s, requests, selenium, pandas
-- Google Chrome
+In this workshop you will learn how to extract data from websites, what you'd call web scraping, using Python. In Episode 1 we begin by reviewing the structure of websites in HTML and how to retrieve information from it using your browser and the `BeautifulSoup` package. In Episode 2 we'll dive deep on how to get the HTML behind any website using the `requests` package and how to parse and find information with `BeautifulSoup`. At the end,you’ll learn about the differences between static and dynamic webpages, and how to scrape the latter with the `Selenium` package.
 
-## Data Sets
+This workshop is designed for participants who already have a basic understanding of Python programming. In particular, it's best to know how to:
 
-<!--
-FIXME: place any data you want learners to use in `episodes/data` and then use
-       a relative link ( [data zip file](data/lesson-data.zip) ) to provide a
-       link to it, replacing the example.com link.
--->
-Download the [data zip file](https://example.com/FIXME) and unzip it to your Desktop
+- Install and import packages and modules
+- Use lists and dictionaries
+- Use conditional statements (`if`, `else`, `elif`)
+- Use `for` loops
+- Calling functions, understanding parameters/arguments and return values
 
 ## Software Setup
 
-::::::::::::::::::::::::::::::::::::::: discussion
-
-### Details
-
-Setup for different systems can be presented in dropdown menus via a `spoiler`
-tag. They will join to this discussion block, so you can give a general overview
-of the software used in this lesson here and fill out the individual operating
-systems (and potentially add more, e.g. online setup) in the solutions blocks.
-
-:::::::::::::::::::::::::::::::::::::::::::::::::::
-
-:::::::::::::::: spoiler
-
-### Windows
-
-Use PuTTY
-
-::::::::::::::::::::::::
-
-:::::::::::::::: spoiler
-
-### MacOS
-
-Use Terminal.app
-
-::::::::::::::::::::::::
-
-
-:::::::::::::::: spoiler
-
-### Linux
-
-Use Terminal
-
-::::::::::::::::::::::::
+Steps:
 
+1. If you already have Anaconda, Jupyter Lab or Jupyter Notebooks installed in your computer, skip to step 2. Follow Miniforge's [download](https://github.com/conda-forge/miniforge?tab=readme-ov-file#download) and [installation](https://github.com/conda-forge/miniforge?tab=readme-ov-file#install) instructions for your respective operating system. If you are using a Windows machine, make sure you mark the option to "Add Miniforge3 to my PATH environment variable".
+2. If you are using Mac or Linux, open the 'Terminal'. If you are using Windows, open the 'Command Prompt' or 'Miniforge Prompt'.
+3. Activate the base conda environment by typing and running the 'conda activate' command.
+4. Install the necessary packages running 'pip install requests beautifulsoup4 selenium webdriver-manager pandas tqdm jupyterlab'.

notebooks/config-conda-env.txt

@@ -1,3 +1,3 @@
 conda create --name webscraping python=3.11.9
 conda activate webscraping
-pip install requests beautifulsoup4 selenium pandas tqdm
+pip install requests beautifulsoup4 selenium webdriver-manager pandas tqdm jupyterlab