{"repo":"blockchain-etl/hedera-etl","free":true,"listed":false,"github":"https://github.com/blockchain-etl/hedera-etl","clone":"git clone https://github.com/blockchain-etl/hedera-etl.git","description":"ETL scripts for Hedera Hashgraph","language":"Java","stars":13,"topics":["hedera","hedera-hashgraph","etl","apache-beam","blockchain-analytics","crypto","cryptocurrency","data-analytics","data-engineering","gcp"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Hedera ETL Hedera ETL populates BigQuery dataset with transactions and records generated by the Hedera Mainnet (or Testnet, if so configured). - Extract: Stream of transactions (and records) are ingested from a Google Cloud Storage Bucket - Transform: Filters for important fields, formats data types, etc - Load: Streaming insert into BigQuery dataset Overview - Record Files contains Protobuf serialized hedera transactions published by Hedera Mirror Node. More details can be found here. - Apache Beam pipeline pulls Record Files from Google Cloud Storage and inserts them into BigQuery. GCP Dataflow is used as runner for the pipeline. - Hedera ETL could be run Setup BigQuery Schema of BigQuery tables is for now managed by Apache Beam. It will be managed by Terraform in near future. Creating tables terraform CLI will be needed to create the tables. See terraform docs how to set them up. ETL to BigQuery Requirements Cloud infrastructure for ETL should exist before deployment. For requirements to deploy on GCP Dataflow, refer deployment. Common parameters Configure GCP project id and cloud infrastructure. To login to your GCP project use: To setup infrastructure see terraform docs. Running from local machine Streaming mode: Batch mode: Extra arguments: - --startAbove= : you can force pipeline to start reading since this, even if it's not the first file that matches your starting point - --disableMergeHistoryInput=true : some records require tracking their state. If you don't requir","default_branch":null,"files":null,"tree":[],"storefront":"/r/blockchain-etl","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/blockchain-etl/hedera-etl/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}