{"repo":"segment-any-text/wtpsplit","free":true,"listed":false,"github":"https://github.com/segment-any-text/wtpsplit","clone":"git clone https://github.com/segment-any-text/wtpsplit.git","description":"Toolkit to segment text into sentences or other semantic units in a robust, efficient and adaptable way.","language":"Python","stars":1330,"topics":["sentence-boundary-detection","python","deep-learning","machine-learning","pretrained-models","sentence-segmentation","sentence-segmenter","natural-language-processing"],"license":"MIT","category":"machine-learning","readme_excerpt":"wtpsplit🪓 Segment any Text - Robustly, Efficiently, Adaptably⚡ This repository allows you to segment text into sentences or other semantic units. It implements the models from: - SaT — Segment Any Text: A Universal Approach for Robust, Efficient and Adaptable Sentence Segmentation by Markus Frohmann, Igor Sterner, Benjamin Minixhofer, Ivan Vulić and Markus Schedl ( state-of-the-art, encouraged ). - WtP — Where’s the Point? Self-Supervised Multilingual Punctuation-Agnostic Sentence Segmentation by Benjamin Minixhofer, Jonas Pfeiffer and Ivan Vulić ( previous version, maintained for reproducibility ). The namesake WtP is maintained for consistency. Our new followup SaT provides robust, efficient and adaptable sentence segmentation across 85 languages at higher performance and less compute cost. Check out the state-of-the-art results in 8 distinct corpora and 85 languages demonstrated in our Segment any Text paper. Installation Or one of the following for ONNX support: Usage ONNX Support 🚀 You can now enable even faster ONNX inference for sat and sat-sm models! 🚀 If you wish to use LoRA in combination with an ONNX model: - Run scripts/export to onnx sat.py with use lora: True and an appropriate output dir: . - If you have a local LoRA module, use lora path . - If you wish to load a LoRA module from the HuggingFace hub, use style or domain and language . - Load the ONNX model with merged LoRA weights: sat = SaT( , onnx providers=[\"CUDAExecutionProvider\", \"CPUExecut","default_branch":null,"files":null,"tree":[],"storefront":"/r/segment-any-text","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/segment-any-text/wtpsplit/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}