{"repo":"IngoKl/textdirectory","free":true,"listed":false,"github":"https://github.com/IngoKl/textdirectory","clone":"git clone https://github.com/IngoKl/textdirectory.git","description":"TextDirectory allows you to filter, transform, and combine multiple text files into one aggregated file.","language":"Python","stars":11,"topics":["python","corpus","corpus-linguistics","text","plaintext","data-science","database","nlp"],"license":"MIT","category":"machine-learning","readme_excerpt":"TextDirectory TextDirectory allows you to combine multiple text files into one aggregated file. TextDirectory also supports matching files for certain criteria and applying transformations to the aggregated text. TextDirectory can be used as a mere tool (via the CLI) and as a Python library. Of course, everything TextDirectory does could be achieved in bash or PowerShell. However, there are certain use cases (e.g., when used as a library) in which it might be useful. Free software: MIT license Documentation: Features Aggregating multiple text files Filtering documents/texts based on various parameters such as length, content, and random sampling Filtering and transforming text files Transforming the aggregated text (e.g., transforming the text to lowercase) Version Filters Transformations --- --- --- 0.1.0 filter by max chars(n int); filter by min chars(n int); filter by max tokens(n int); filter by min tokens(n int); filter by contains(str); filter by not contains(str); filter by random sampling(n int; replace=False) transformation lowercase 0.1.1 filter by chars outliers(n sigmas int) transformation remove nl 0.1.2 filter by filename contains(contains str) transformation usas en semtag; transformation uppercase; transformation postag(spacy model str) 0.1.3 filter by similar documents(reference file str; threshold float) transformation remove non ascii; transformation remove non alphanumerical 0.2.0 filter by max filesize(max kb int); filter by min filesize(min kb int) trans","default_branch":null,"files":null,"tree":[],"storefront":"/r/IngoKl","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/IngoKl/textdirectory/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}