{"repo":"datagouv/csv-detective","free":true,"listed":false,"github":"https://github.com/datagouv/csv-detective","clone":"git clone https://github.com/datagouv/csv-detective.git","description":"Inspection of tabular (csv, xls-like, parquet) files to guess the columns' content","language":"Python","stars":53,"topics":["data-analysis","data-engineering","government-data","open-data"],"license":"MIT","category":"data-pipelines","readme_excerpt":"CSV Detective This is a package to automatically detect column content in tabular files . The script reads either the whole file or the first few rows and performs various checks (regex, casting, comparison with official lists...) to see for each column if it matches with various content types. Currently supported file types: csv(.gz), xls, xlsx, ods, parquet. You can also directly feed the URL of a remote file (from data.gouv.fr for instance). How To? Install the package You need to have Python = 3.11 installed. We recommend using a virtual environment. Detect some columns Say you have a tabular file located at file path . This is how you could use csv detective : So What Do You Get ? Output The program creates a python dictionary with the following information : The output slightly differs depending on the file format: - csv files have encoding and separator (and compression if relevant) - xls, xlsx, ods files have engine and sheet name - parquet only has engine You may also set output df to True , in which case the output is a tuple of two elements: - the analysis (as described above) - an iterator of pd.DataFrame s which contain the columns cast with the detected types (which can be used with pd.concat or in a loop): What Formats Can Be Detected Includes : - types (float, int, dates, datetimes, JSON) and more specific (latitude, longitude, geoJSON...) - Communes, Départements, Régions, Pays - Codes Communes, Codes Postaux, Codes Departement, ISO Pays - Codes CSP, Descript","default_branch":null,"files":null,"tree":[],"storefront":"/r/datagouv","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/datagouv/csv-detective/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}