{"repo":"code4craft/webmagic","free":true,"listed":false,"github":"https://github.com/code4craft/webmagic","clone":"git clone https://github.com/code4craft/webmagic.git","description":"A scalable web crawler framework for Java.","language":"Java","stars":11680,"topics":["crawler","java","scraping","framework"],"license":"Apache-2.0","category":"scrapers-browser-automation","readme_excerpt":"Readme in Chinese A scalable crawler framework. It covers the whole lifecycle of crawler: downloading, url management, content extraction and persistent. It can simplify the development of a specific crawler. Features: Simple core with high flexibility. Simple API for html extracting. Annotation with POJO to customize a crawler, no configuration. Multi-thread and Distribution support. Easy to be integrated. Install: Add dependencies to your pom.xml: WebMagic use slf4j with slf4j-log4j12 implementation. If you customized your slf4j implementation, please exclude slf4j-log4j12. Get Started: First crawler: Write a class implements PageProcessor. For example, I wrote a crawler of github repository information. page.addTargetRequests(links) Add urls for crawling. You can also use annotation way: Docs and samples: Documents: http://webmagic.io/docs/ The architecture of webmagic (referred to Scrapy) There are more examples in webmagic-samples package. License: Licensed under Apache 2.0 license Thanks: To write webmagic, I refered to the projects below : Scrapy A crawler framework in Python. http://scrapy.org/ Spiderman Another crawler framework in Java. http://git.oschina.net/l-weiwei/spiderman Mail-list: https://groups.google.com/forum/#!forum/webmagic-java http://list.qq.com/cgi-bin/qf invite?id=023a01f505246785f77c5a5a9aff4e57ab20fcdde871e988 QQ Group: 373225642 542327088 Related Project Gather Platform A web console based on WebMagic for Spider configuration and management.","default_branch":null,"files":null,"tree":[],"storefront":"/r/code4craft","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/code4craft/webmagic/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}