{"repo":"codelibs/fess-crawler","free":true,"listed":false,"github":"https://github.com/codelibs/fess-crawler","clone":"git clone https://github.com/codelibs/fess-crawler.git","description":"Web/FileSystem Crawler Library","language":"Java","stars":39,"topics":["crawler","java"],"license":"Apache-2.0","category":"scrapers-browser-automation","readme_excerpt":"Fess Crawler Fess Crawler is a Java crawling framework for enterprise-scale content acquisition and text extraction. It is the crawling engine behind Fess, and can be embedded in any JVM application on its own. Key Features - Multi-protocol : HTTP/HTTPS, local/network file systems, FTP/FTPS, SMB/CIFS (SMB1 and SMB2+), MinIO-compatible object storage, Amazon S3, Google Cloud Storage - Broad text extraction : Office documents, PDF, HTML, XML, JSON, CSV, Markdown, e-mail, archives, PostScript, and media/image metadata — via Apache Tika, POI, PDFBox and format-specific extractors - Multi-threaded crawling with configurable thread pools, depth limits and access-count limits - Fault tolerant : retry wrapper ( FaultTolerantClient ) and an HTTP client that can fall back between Apache HttpComponents 5.x and 4.x - Polite : robots.txt (RFC 9309) and sitemap support, plus configurable request intervals - Bounded by design : content-length caps enforced while downloading, archive-bomb and Zip Slip defenses, and bounded recursion in nested extractors - Pluggable : clients, extractors, transformers, rules and filters are all replaceable via DI - Pluggable storage : in-memory by default, OpenSearch for persistent/distributed crawls Requirements - Java 21 or later - Maven 3.8 or later (to build from source) Installation Releases are published to the CodeLibs Maven repository rather than Maven Central, so declare it first. Browse the artifact directory for the current release. Then add the mo","default_branch":null,"files":null,"tree":[],"storefront":"/r/codelibs","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/codelibs/fess-crawler/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}