{"repo":"calbergs/spotify-api","free":true,"listed":false,"github":"https://github.com/calbergs/spotify-api","clone":"git clone https://github.com/calbergs/spotify-api.git","description":"Pipeline that extracts data from the Spotify API to build a more detailed version of Spotify Wrapped","language":"Python","stars":50,"topics":["airflow","dbt","python","data-engineering","metabase","postgresql","docker"],"license":null,"category":"data-pipelines","readme_excerpt":"Spotify Data Pipeline Data pipeline that extracts a user's song listening history from the Spotify API using Python, PostgreSQL, dbt, Metabase, Airflow, and Docker Objective Deep dive into a user's song listening history to retrieve information about top artists, top tracks, top genres, and more. This is a personal side project for fun to recreate Spotify Wrapped but at a more frequent cadence to get quicker and more detailed insights. This pipeline calls the Spotify API every hour from hours 0-6 and 14-23 UTC (basically whenever I'm awake) to extract a user's song listening history, load the responses into a database, apply transformations and visualize the metrics in a dashboard. Since the dataset is small and this doesn't need to be running 24/7 this is all built using open source tools and hosted locally to avoid any cost. Tools & Technologies - Containerization - Docker , Docker Compose - Orchestration - Airflow - Database - PostgreSQL - Transformation - dbt - Data Visualization - Metabase - Language - Python Architecture Data Flow 1. main.py script is triggered every hour (from hours 0-6 and 14-23 UTC) via Airflow to refresh the access token, make a connection to the Postgres database to check for the latest listened time, and call the Spotify API to retrieve the most recently played songs and corresponding genres. 2. Responses are saved as CSV files in 'YYYY-MM-DD.csv' format. These are saved on the local file system and act as our replayable source since the Spotify A","default_branch":null,"files":null,"tree":[],"storefront":"/r/calbergs","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/calbergs/spotify-api/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}