{"repo":"ScrapeOps/scrapeops-scrapy-sdk","free":true,"listed":false,"github":"https://github.com/ScrapeOps/scrapeops-scrapy-sdk","clone":"git clone https://github.com/ScrapeOps/scrapeops-scrapy-sdk.git","description":"Scrapy extension that gives you all the scraping monitoring, alerting, scheduling, and data validation you will need straight out of the box.","language":"Python","stars":38,"topics":["scrapy","scrapyd","monitoring","spider","scrapyd-ui","scrapy-visualization","scrapy-monitor","scrapy-log-analysis","scraping","monitoring-tool"],"license":"BSD-3-Clause","category":"scrapers-browser-automation","readme_excerpt":"ScrapeOps Scrapy SDK: Scrapy Extension For Spider Monitoring, Alerts and Scheduling. The ScrapeOps Scrapy SDK is an extension for your Scrapy spiders that gives you all the scraping monitoring, statistics, alerting, scheduling and data validation you will need straight out of the box. Just enable it in your settings.py file and the SDK will automatically monitor your scrapers and send your logs to your scraping dashboard. When connected to a ScrapyD server, you can schedule and manage all your jobs from one easy to use interface. Full documentation can be found here: ScrapeOps Documentation :computer: Demo :link: ScrapeOps Dashboard Demo :star: Features View features - Scrapy Job Stats & Visualisation - :chart with upwards trend: Individual Job Progress Stats - :bar chart: Compare Jobs versus Historical Jobs - :100: Job Stats Tracked - :white check mark: Pages Scraped & Missed - :white check mark: Items Parsed & Missed - :white check mark: Item Field Coverage - :white check mark: Runtimes - :white check mark: Response Status Codes - :white check mark: Success Rates & Average Latencies - :white check mark: Errors & Warnings - :white check mark: Bandwidth - Health Checks & Alerts - :male detective: Custom Spider & Job Health Checks - :package: Out of the Box Alerts - Slack (More coming soon!) - :bookmark tabs: Daily Scraping Reports - ScrapyD Cluster Management - :link: Integrate With ScrapyD Servers - :alarm clock: Schedule Periodic Jobs - :100: All Scrapyd JSON API Supported ","default_branch":null,"files":null,"tree":[],"storefront":"/r/ScrapeOps","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/ScrapeOps/scrapeops-scrapy-sdk/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}