{"repo":"jfilter/clean-text","free":true,"listed":false,"github":"https://github.com/jfilter/clean-text","clone":"git clone https://github.com/jfilter/clean-text.git","description":"🧹 Python package for text cleaning","language":"Python","stars":1025,"topics":["python","natural-language-processing","text-cleaning","text-normalization","text-preprocessing","python-package","nlp","user-generated-content","scraping"],"license":null,"category":"scrapers-browser-automation","readme_excerpt":"clean-text User-generated content on the Web and in social media is often dirty. Preprocess your scraped data with clean-text to create a normalized text representation. For instance, turn this corrupted input: into this clean output: clean-text uses ftfy, unidecode and numerous hand-crafted rules, i.e., RegEx. Installation To install the GPL-licensed package unidecode alongside: You may want to abstain from GPL: NB: This package is named clean-text and not cleantext . If unidecode is not available, clean-text will resort to Python's unicodedata.normalize for transliteration. Transliteration to closest ASCII symbols involes manually mappings, i.e., ê to e . unidecode 's mapping is superiour but unicodedata's are sufficent. However, you may want to disable this feature altogether depending on your data and use case. To make it clear: There are inconsistencies between processing text with or without unidecode . Usage Carefully choose the arguments that fit your task. The default parameters are listed above. Preserving patterns with exceptions Use exceptions to protect specific text patterns from being modified during cleaning. Each entry is a regex pattern string; all matches are preserved verbatim (not lowered, not transliterated — exactly as they appeared in the input). You may also only use specific functions for cleaning. For this, take a look at the source code. Cleaning multiple texts in parallel Use clean texts() to clean a list of strings. Set n jobs to enable parallel ","default_branch":null,"files":null,"tree":[],"storefront":"/r/jfilter","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/jfilter/clean-text/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}