{"repo":"tasos-py/Search-Engines-Scraper","free":true,"listed":false,"github":"https://github.com/tasos-py/Search-Engines-Scraper","clone":"git clone https://github.com/tasos-py/Search-Engines-Scraper.git","description":"Search google, bing, yahoo, and other search engines with python","language":"Python","stars":671,"topics":["python","search-engine","scraper","google","bing","yahoo","crawler"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"search engines A Python library that queries Google, Bing, Yahoo and other search engines and collects the results from multiple search engine results pages. Please note that web-scraping may be against the TOS of some search engines, and may result in a temporary ban. Supported search engines Google Bing Yahoo Duckduckgo Startpage Aol Dogpile Ask Mojeek Brave Torch Features - Creates output files (html, csv, json). - Supports search filters (url, title, text). - HTTP and SOCKS proxy support. - Collects dark web links with Torch. - Easy to add new search engines. You can add a new engine by creating a new class in search engines/engines/ and add it to the search engines dict dictionary in search engines/engines/ init .py . The new class should subclass SearchEngine , and override the following methods: selectors , first page , next page . - Python2 - Python3 compatible. Requirements Python 2.7 - 3.x with Requests and BeautifulSoup Installation Run the setup file: $ python setup.py install . Done! Usage As a library: As a CLI script: Other versions - async-search-scraper A really cool asynchronous implementation, written by @soxoj","default_branch":null,"files":null,"tree":[],"storefront":"/r/tasos-py","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/tasos-py/Search-Engines-Scraper/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}