{"repo":"AtilioA/web-scraping-for-sentence-mining","free":true,"listed":false,"github":"https://github.com/AtilioA/web-scraping-for-sentence-mining","clone":"git clone https://github.com/AtilioA/web-scraping-for-sentence-mining.git","description":"⛏️ Scraping the web to automate creation of Anki cards.","language":"Python","stars":11,"topics":["anki-cards","anki-deck","anki-flashcards","web-scraping","language-learning"],"license":"Unlicense","category":"scrapers-browser-automation","readme_excerpt":"web-scraping-for-sentence-mining This repository contains scripts designed to scrape content from various websites for language learning. While most of the websites targeted are in my native language, the repository also includes support for Reverso Context, which works with multiple languages. These scripts download or generate audio, and then write all the scraped data to CSV files. You can then merge these files and import them into Anki, allowing you to create thousands of language-learning flashcards in just a few minutes. See also: Morphman - for dealing with low-yield, repetitive, cards too short/long. 🏡 Running locally First, clone the repository and create a virtual environment for the project (to ensure you won't have problems with your libraries' versions). Using virtualenvwrapper: mkvirtualenv web-scraping-sentences - Creates a virtual environment workon web-scraping-sentences - Activates virtual environment Then, install all dependencies with pip install -r requirements.txt . Additional Setup for WaveNet If you want to use WaveNet, you need to get your key on Google Cloud Platform and fill it in api key.json , following the example from api key example.json . Scraping with Reverso To scrape Reverso, head to reverso scraping/ and refer to scrap.py . Input URLs you want to scrape into scrap page() . By default, the output will be placed under audios/ for WaveNet audios and under csv/ for csv files (the tab character, \\t, is used as separator) ready to be imported ","default_branch":null,"files":null,"tree":[],"storefront":"/r/AtilioA","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/AtilioA/web-scraping-for-sentence-mining/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}