{"repo":"oxylabs/Scraping-Dynamic-JavaScript-Ajax-Websites-With-BeautifulSoup","free":true,"listed":false,"github":"https://github.com/oxylabs/Scraping-Dynamic-JavaScript-Ajax-Websites-With-BeautifulSoup","clone":"git clone https://github.com/oxylabs/Scraping-Dynamic-JavaScript-Ajax-Websites-With-BeautifulSoup.git","description":"A guide on how to scrape JavaScript rendered websites with Python and BeautifulSoup.","language":"Python","stars":51,"topics":["javascript","ajax","beautiful-soup","web-scraping","python","scraping","github-python"],"license":null,"category":"scrapers-browser-automation","readme_excerpt":"Scraping Dynamic JavaScript / Ajax Websites With BeautifulSoup: A Complete Tutorial Table of contents - Revisiting BeautifulSoup and Requests - Is This Website Dynamic or Static? - Can BeautifulSoup Render JavaScript ? - Scraping Dynamic Web Pages With Selenium - Finding Elements Using Selenium - Finding Elements Using BeautifulSoup - Headless browser - Web Scraping Dynamic Sites by Locating AJAX Calls - Data Embedded In the Same Page - Data In Other Pages Web scraping most of the websites may be comparatively easy. This topic is already covered at length in this tutorial. There are many sites, however, which can not be scraped using the same method. The reason is that these sites load the content dynamically using JavaScript. This technique is also known as AJAX (Asynchronous JavaScript and XML). Historically, this standard was included creating an XMLHttpRequest object to retrieve XML from a web server without reloading the whole page. These days, this object is rarely used directly. Usually, a wrapper like jQuery is used to retrieve content such as JSON, partial HTML, or even images. Revisiting BeautifulSoup and Requests To scrape a regular web page, at least two libraries are required. The requests library downloads the page. Once this page is available as an HTML string, the next step is parsing this as a BeautifulSoup object. This BeautifulSoup object can then be used to find specific data. Here is a simple example script that prints the text inside the h1 element with ","default_branch":null,"files":null,"tree":[],"storefront":"/r/oxylabs","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/oxylabs/Scraping-Dynamic-JavaScript-Ajax-Websites-With-BeautifulSoup/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}