{"repo":"mushroomsandchai/eia","free":true,"listed":false,"github":"https://github.com/mushroomsandchai/eia","clone":"git clone https://github.com/mushroomsandchai/eia.git","description":"Data pipeline for EIA datasets.","language":"Python","stars":12,"topics":["airflow","bigquery","dbt"],"license":null,"category":"data-pipelines","readme_excerpt":"EIA Data Pipeline A data pipeline for Energy Information Administration (EIA) data using Apache Airflow, dbt, cosmos and PostgreSQL with infrastructure as code via Terraform. Problem Description The U.S. Energy Information Administration (EIA) publishes near real-time and historical data on electricity generation, grid demand, and cross-border energy flows across the United States. While this data is freely accessible via the EIA API, it is returned in a raw, fragmented format, split across multiple endpoints, inconsistently typed, and lacking the joins and aggregations needed to draw meaningful conclusions. The problem this project solves is the absence of a reliable, automated pipeline that continuously ingests this data, transforms it into a consistent and analytics ready format, and makes it accessible for visual exploration. Without such a pipeline, answering fundamental questions about the U.S. electricity grid requires significant manual effort each time: - How has the fuel mix shifted over time, are renewables actually displacing fossil fuels? - Is electricity demand being forecast accurately by grid operators? - How much electricity flows between the U.S., Canada, and Mexico, and in which direction? This project addresses that gap by building an end-to-end data pipeline that ingests raw EIA data via Airflow (both in bulk from pre-exported Parquet files and incrementally via the live API), transforms it through a layered dbt model into clean, joined, and aggregated ta","default_branch":null,"files":null,"tree":[],"storefront":"/r/mushroomsandchai","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/mushroomsandchai/eia/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}