{"repo":"feyninc/pulpie","free":true,"listed":false,"github":"https://github.com/feyninc/pulpie","clone":"git clone https://github.com/feyninc/pulpie.git","description":"Pareto-optimal models for cleaning the web — fast, encoder-based main-content extraction from HTML.","language":"HTML","stars":67,"topics":["boilerplate-removal","content-extraction","encoder","html-extraction","html-to-markdown","nlp","rag","transformers","web-scraping","main-content-extraction"],"license":"Apache-2.0","category":"machine-learning","readme_excerpt":"Pareto-optimal models for cleaning the web. Extract main content from HTML at one twentieth the cost. Install • Usage • Models • How it works • Benchmarks • Blog Pulpie extracts the main content from raw HTML, stripping navigation, ads, sidebars, and footers. It uses small encoder models that label every block in a single forward pass, approaching state-of-the-art extraction quality while running up to 20x faster and 20x cheaper than autoregressive extractors on an L4 GPU. - Fast. An encoder labels every block in one forward pass (13.7 pages/sec on an L4). - Accurate. Matches state-of-the-art quality: 0.862 to 0.873 ROUGE-5 F1 on WebMainBench. - Small. The recommended model is 210M parameters and fits on any GPU. - Cheap. Clean 1 billion pages for $7,900, versus $159,000 for the leading decoder. - Simple. Run pip install pulpie , then Extractor().extract(html) . - Batched. An overlapped CPU and GPU pipeline scales across multiple GPUs. Installation For Markdown output, install the markdown extra: Or with uv: Usage Basic The model downloads from Hugging Face on first use. Choosing a model Batch processing For bulk extraction, Pipeline overlaps CPU preprocessing with GPU inference and self-balances across one or more GPUs: Models All three models are built on EuroBERT, share a tokenizer, and use the same block-marker architecture. Large is the teacher; Base and Small are distilled from it. Model Hugging Face Params ROUGE-5 F1 Notes ------- -------------- -------- ------------ -","default_branch":null,"files":null,"tree":[],"storefront":"/r/feyninc","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/feyninc/pulpie/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}