{"repo":"oxylabs/web-scraping-data-parsing-beautiful-soup","free":true,"listed":false,"github":"https://github.com/oxylabs/web-scraping-data-parsing-beautiful-soup","clone":"git clone https://github.com/oxylabs/web-scraping-data-parsing-beautiful-soup.git","description":"Web Scraping and Data Parsing Using Beautiful Soup","language":"Python","stars":12,"topics":["beautifulsoup","data-parsing","python","web-scraping","github-python"],"license":null,"category":"scrapers-browser-automation","readme_excerpt":"Web Scraping and Data Parsing Using Beautiful Soup This project provides a clear and concise example of how to fetch content from a website using the Requests module and then parse it using BeautifulSoup. Setting Up To run this example you will need Python 3. We recommend setting up a virtual environment Install dependencies by running Note : You can also install them by using the requirements.txt file included in this repository. Web Scraping A mock bookstore website called https://books.toscrape.com is our scraping target. Use the requests module to fetch a page from it Once the response is retrieved, check whether the request was successful or not by verifying the status code property Save the script as src/scrape.py and run it. The requests module has successfully retrieved the html content from the website and now all that's left is to parse it. A working example can be found here Parse HTML Take a look at the structure of the HTML that you're trying to scrape. The book info is neatly wrapped in an article tag. Inside the article, there's a heading ( h3 ) that contains an anchor ( a ), which contains the title of the book inside an attribute. To parse this HTML content use the BeautifulSoup4 library. Firstly, import BeautifulSoup Then, create an instance of the BeautifulSoup class and load the HTML content that has been retrieved from the web page previously. Retrieve all the article tags Define a titles array that will hold all the book titles extracted from the current","default_branch":null,"files":null,"tree":[],"storefront":"/r/oxylabs","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/oxylabs/web-scraping-data-parsing-beautiful-soup/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}