{"repo":"blockchain-etl/ethereum-etl-airflow","free":true,"listed":false,"github":"https://github.com/blockchain-etl/ethereum-etl-airflow","clone":"git clone https://github.com/blockchain-etl/ethereum-etl-airflow.git","description":"Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. How to get any Ethereum smart contract into BigQuery https://towardsdatascience.com/how-to-get-any-ethereum-smart-contract-into-bigquery-in-8-mins-bab5db1fdeee","language":"Python","stars":440,"topics":["ethereum","blockchain-analytics","data-engineering","cryptocurrency","web3","apache-airflow","crypto","data-analytics","etl","gcp"],"license":"MIT","category":"blockchain-web3","readme_excerpt":"Ethereum ETL Airflow Read this article: https://cloud.google.com/blog/products/data-analytics/ethereum-bigquery-how-we-built-dataset Local Development Prerequisites - direnv - pyenv We are using direnv to automatically set up and load the correct python version. We also create a venv in the root folder, that is automatically activated when entering the project folder. Setting up Airflow DAGs using Google Cloud Composer Create BigQuery Datasets - Sign in to BigQuery https://bigquery.cloud.google.com/ - Create new datasets called crypto ethereum , crypto ethereum raw , crypto ethereum temp Create Google Cloud Storage bucket - Create a new Google Storage bucket to store exported files https://console.cloud.google.com/storage/browser Create Google Cloud Composer (version 2) environment Create a new Cloud Composer environment: Create variables in Airflow ( Admin Variables in the UI): Variable Description ----------------------------------------- ----------------------------------------- ethereum output bucket GCS bucket to store exported files ethereum provider uris Comma separated URIs of Ethereum nodes ethereum destination dataset project id Project ID of BigQuery datasets notification emails email for notifications Check other variables in dags/ethereumetl airflow/variables.py . Updating package requirements Suggested package requirements for Composer are stored in requirements airflow.txt . You can update the Composer environment using the following script: Note: Composer can ","default_branch":null,"files":null,"tree":[],"storefront":"/r/blockchain-etl","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/blockchain-etl/ethereum-etl-airflow/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}