{"repo":"Murrough-Foley/rs-trafilatura","free":true,"listed":false,"github":"https://github.com/Murrough-Foley/rs-trafilatura","clone":"git clone https://github.com/Murrough-Foley/rs-trafilatura.git","description":"Fast, accurate web content extraction in Rust. ML page-type classification, per-type extraction, confidence scoring. F1=0.966 on ScrapingHub (#1), F1=0.859 across 2,008 annotated pages (1,497 development + 511 held-out test","language":"Rust","stars":50,"topics":["content-extraction","rust","search-engine-optimization","trafilatura","web-scraping","machine-learning","nlp"],"license":"Apache-2.0","category":"machine-learning","readme_excerpt":"rs-trafilatura Fast and accurate web content extraction in Rust. A high-performance Rust port of trafilatura / go-trafilatura, extracting clean, readable content from web pages while removing boilerplate, navigation, and advertisements. Features - Fast : 71 files/s for articles, 46 files/s overall on a 1,497-page benchmark (pure Rust, compile-time regex) - Accurate : F1 0.966 on ScrapingHub benchmark, F1 0.859 across 7 page types - Page Type Classification : XGBoost classifier (200 trees, 181 features) detects 7 page types: article, forum, product, collection, listing, documentation, service - Per-Type Extraction : Specialized extraction profiles tuned for each page type (12 forum platforms, 4 documentation frameworks, JSON-LD product fallback) - Extraction Quality Predictor : ML-based confidence scoring (0.0-1.0) using a 27-feature XGBoost model that predicts extraction F1 — pages below 0.80 are candidates for LLM fallback - Markdown Output : GitHub Flavored Markdown preserving headings, lists, tables, bold/italic, code blocks - Rich Metadata : Title, author, date, description, categories, tags, license, images from JSON-LD, Open Graph, Dublin Core, and HTML meta tags - Configurable : 28 options to tune precision/recall tradeoff, content selection, and output format - Robust : Handles malformed HTML gracefully with automatic character encoding detection (UTF-8, ISO-8859-1, Windows-1252) Quick Start Installation Add to your Cargo.toml : Usage Basic Extraction Custom Options M","default_branch":null,"files":null,"tree":[],"storefront":"/r/Murrough-Foley","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Murrough-Foley/rs-trafilatura/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}