{"repo":"toddwschneider/nyc-taxi-data","free":true,"listed":false,"github":"https://github.com/toddwschneider/nyc-taxi-data","clone":"git clone https://github.com/toddwschneider/nyc-taxi-data.git","description":"Import public NYC taxi and for-hire vehicle (Uber, Lyft) trip data into a PostgreSQL or ClickHouse database","language":"R","stars":2080,"topics":["clickhouse","nyc","nyc-taxi-dataset","postgresql"],"license":"MIT","category":"databases-storage","readme_excerpt":"New York City Taxi and For-Hire Vehicle Data Scripts to download, process, and analyze data from 3+ billion taxi and for-hire vehicle (Uber, Lyft, etc.) trips originating in New York City since 2009. There are separate sets of scripts for storing data in either a PostgreSQL or ClickHouse database. Most of the raw data comes from the NYC Taxi & Limousine Commission. The repo was created originally in support of this post: Analyzing 1.1 Billion NYC Taxi and Uber Trips, with a Vengeance TLC 2022 Parquet Format Update The TLC changed the raw data format from CSV to Apache Parquet in May 2022, including a full replacement of all historical files. This repo is now updated to handle the Parquet files in one of two ways: 1. The \"old\" Postgres-based code still works, by adding an intermediate step that converts each Parquet file into a CSV before using the Postgres COPY command 2. A separate set of scripts loads the Parquet files directly into a ClickHouse database As part of the May 2022 update, the TLC added several new columns to the High Volume For-Hire Vehicle (Uber, Lyft) trip files, including information about passenger fares, driver pay, and time spent waiting for passengers. These new fields are available back to February 2019. This repo no longer works with the old CSV files provided by the TLC. Those files are no longer available to download from the TLC's website, but if you happen to have them lying around and want to use this repo, you should look at this older version o","default_branch":null,"files":null,"tree":[],"storefront":"/r/toddwschneider","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/toddwschneider/nyc-taxi-data/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}