{"repo":"bahaeddinmselmi/derja-smart-scraper","free":true,"listed":false,"github":"https://github.com/bahaeddinmselmi/derja-smart-scraper","clone":"git clone https://github.com/bahaeddinmselmi/derja-smart-scraper.git","description":"A lightweight CLI tool for collecting Tunisian Derja text snippets from the open web. It queries Google via [SerpAPI](https://serpapi.com), downloads each result, extracts readable text, and keeps only the sentences that look like Tunisian Derja using a heuristic detector.","language":"Python","stars":10,"topics":["dataset-creation","nlp","python","serpapi","tunisian-arabic","web-scraping"],"license":"MIT","category":"scrapers-browser-automation","readme_excerpt":"🇹🇳 Derja Smart Web Scraper A specialized web mining tool for Tunisian Arabic (Derja) , designed to build high-quality NLP datasets for LLM training. 📖 Overview Derja Smart Web Scraper is a lightweight CLI utility that automates the collection of Tunisian Arabic text from the open web. It leverages Google Search (via SerpAPI) to discover authentic content, scrapes the resulting pages, and applies a heuristic Derja Detector to filter for relevant sentences. It is specifically engineered to handle the nuances of the Tunisian dialect, including: Code-switching : Mixing French/English with Arabic. Arabizi : Latin-script Arabic (e.g., \"chnowa a7welek\"). Arabic Script : Standard and dialectal Arabic text. ✨ Key Features Intelligent Search : Automates Google queries to find niche forums and blogs. Smart Filtering : Uses a vocabulary-based scoring system ( score derja ) to distinguish Tunisian Derja from MSA (Modern Standard Arabic) or other noise. Clean Output : Produces ready-to-train JSONL data, compatible with standard NLP pipelines. Robust Cleaning : Strips HTML, ads, and navigation elements using BeautifulSoup. 🛠 Installation 1. Clone the repository : 2. Install dependencies : 3. Set up API Key : You need a SerpAPI key to perform searches. 🚀 Usage Run the scraper with custom queries to start building your dataset: Options Flag Description Default ------ ------------- --------- --queries List of search terms (Internal List) --min score Threshold for Derja detection (0.0 - 1.","default_branch":null,"files":null,"tree":[],"storefront":"/r/bahaeddinmselmi","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/bahaeddinmselmi/derja-smart-scraper/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}