{"repo":"mykkelol/netsuite-data-pipeline","free":true,"listed":false,"github":"https://github.com/mykkelol/netsuite-data-pipeline","clone":"git clone https://github.com/mykkelol/netsuite-data-pipeline.git","description":"📈 Finance Data Pipeline to extract all GL posting transactions from NetSuite to S3 and Postgres for analysis, compliance, and automation","language":"Python","stars":10,"topics":["airflow","data-pipeline","etl","netsuite","python","s3","sql","apache-airflow","automation","dag"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"Finance Data Pipeline Finance Data Pipeline is a purpose-built data pipeline to help Finance teams in complying with the 2023 mandate from the India Ministry of Corporate Affairs (👉 see MCA Notification 21st August 2022) without increasing manual workload by automatically orchestrating large volume of financial data across multiple systems. The mandate requires all enterprises operating in India, including branches of US-based companies, to maintain daily backups of financial data. The data pipeline ensures that Finance teams adhere to evolving accounting standards and regulations by leveraging parallel programming, REST APIs, and incremental load ELT architecture to support the following stack: - ERP : NetSuite, SuiteScript - Orchestrator : Airflow - Data storage : Postgres, AWS S3 - Analytics : Tableau, Mode, Adaptive Planning - Internal Tool : Retool Architecture How it works Finance Data Pipeline facilitates Finance teams' compliance requirements without increasing manual work. It leverages Airflow, parallel programming, and an incremental load ELT architecture to automatically: 1. Extract data from NetSuite ERP by executing a pool of DAG tasks hourly to call a custom NetSuite SuiteScript API to extract transactions and related metadata such as vendors, customers, employees, departments, files, etc. asynchronously and dynamically. 2. Load data into SQL warehouse and India-hosted S3 lake. 3. Transform data to structured, audit-ready, and reportable data, empowering Financ","default_branch":null,"files":null,"tree":[],"storefront":"/r/mykkelol","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/mykkelol/netsuite-data-pipeline/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}