{"repo":"edwindj/chunked","free":true,"listed":false,"github":"https://github.com/edwindj/chunked","clone":"git clone https://github.com/edwindj/chunked.git","description":"Chunkwise Text-file Processing for 'dplyr'","language":"R","stars":168,"topics":["dplyr","chunk","database","r"],"license":null,"category":"databases-storage","readme_excerpt":"chunked R is a great tool, but processing data in large text files is cumbersome. chunked helps you to process large text files with dplyr while loading only a part of the data in memory. It builds on the excellent R package LaF . Processing commands are written in dplyr syntax, and chunked (using LaF ) will take care that chunk by chunk is processed, taking far less memory than otherwise. chunked is useful for select -ing columns, mutate -ing columns and filter -ing rows. It is less helpful in group -ing and summarize -ation of large text files. It can be used in data pre-processing. Install ‘chunked’ can be installed with beta version with: and the development version with: Enjoy! Feedback is welcome… Usage Text file - process - text file Most common case is processing a large text file, select or add columns, filter it and write the result back to a text file chunked will write process the above statement in chunks of 5000 records. This is different from for example read.csv which reads all data into memory before processing it. Text file - process - database Another option is to use chunked as a preprocessing step before adding it to a database Db - process - Text file Chunked can be used to export chunkwise to a text file. Note however that in that case processing takes place in the database and the chunkwise restrictions only apply to the writing. Lazy processing chunked will not start processing until collect or write chunkwise is called. Syntax completion of variables","default_branch":null,"files":null,"tree":[],"storefront":"/r/edwindj","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/edwindj/chunked/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}