{"repo":"rat-nick/film-data-ingestion-pipeline","free":true,"listed":false,"github":"https://github.com/rat-nick/film-data-ingestion-pipeline","clone":"git clone https://github.com/rat-nick/film-data-ingestion-pipeline.git","description":"A repository defining a simple data pipleine for ETL jobs relating to media metadata.","language":"Python","stars":52,"topics":["airflow","data-engineering","docker","docker-compose","etl-pipeline","example-project"],"license":"BSD-3-Clause","category":"data-pipelines","readme_excerpt":"Film Data Ingestion Pipeline This project is designed to fetch, process, and store film metadata from The Movie Database (TMDB) using Apache Airflow. The pipeline consists of several components including data fetching, processing, and loading into a PostgreSQL database. Project Structure - dags/ : Contains the Airflow DAGs. - weekly film discovery dag.py : Defines the DAG for discovering and processing films on a weekly basis. - dags/utils/ : Contains utility scripts. - films.py : Contains functions to fetch film data and genres from TMDB. - docker-compose.yaml : Docker Compose configuration to set up the Airflow and PostgreSQL services. - schema.sql : SQL schema for the database containing the production data. Getting Started Prerequisites - Docker - Docker Compose - TMDB API key Setup 1. Clone the repository: 2. Create a .env and run the following commands: 3. Start the services using Docker Compose: 4. Access the Airflow webserver at http://localhost:8080 and log in with the default credentials ( airflow / airflow ). 5. Add a new connection from the Admin tab. You should use the following parameters: Parameter Value --------------- ------------------- Connection Id media-metadata-db Connection Type Postgres Host media-db Schema media Login media Password media Port 5432 Of course, you could use a DBaaS like neon, but the free tier is limited for the data that will be ingested. Just don't forget to change the connection parameters. 6. Enable the weekly film discovery DAG an","default_branch":null,"files":null,"tree":[],"storefront":"/r/rat-nick","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/rat-nick/film-data-ingestion-pipeline/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}