{"repo":"dmi3kno/polite","free":true,"listed":false,"github":"https://github.com/dmi3kno/polite","clone":"git clone https://github.com/dmi3kno/polite.git","description":"Be nice on the web","language":"R","stars":334,"topics":["robotstxt","crawler","webscraping","scraper","r","rstats","r-package","memoise","rate-limiter","rvest"],"license":null,"category":"scrapers-browser-automation","readme_excerpt":"polite The goal of polite is to promote responsible web etiquette. “bow and scrape” (verb): 1) To make a deep bow with the right leg drawn back (thus scraping the floor), left hand pressed across the abdomen, right arm held aside. 2) (idiomatic, by extension) To behave in a servile, obsequious, or excessively polite manner. \\[1\\] Source: Wiktionary, The free dictionary The package’s two main functions bow and scrape define and realize a web harvesting session. bow is used to introduce the client to the host and ask for permission to scrape (by inquiring against the host’s robots.txt file), while scrape is the main function for retrieving data from the remote server. Once the connection is established, there’s no need to bow again. Rather, in order to adjust a scraping URL the user can simply nod to the new path, which updates the session’s URL, making sure that the new location can be negotiated against robots.txt . The three pillars of a polite session are seeking permission, taking slowly and never asking twice . The package builds on awesome toolkits for defining and managing http sessions ( httr and rvest ), declaring the user agent string and investigating site policies ( robotstxt ), and utilizing rate-limiting and response caching ( ratelimitr and memoise ). Installation You can install polite from CRAN with: Development version of the package can be installed from Github with: Basic Example This is a basic example which shows how to retrieve the list of semi-soft chee","default_branch":null,"files":null,"tree":[],"storefront":"/r/dmi3kno","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/dmi3kno/polite/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}