{"repo":"Trendyol/Transporter","free":true,"listed":false,"github":"https://github.com/Trendyol/Transporter","clone":"git clone https://github.com/Trendyol/Transporter.git","description":"Transporter is an open-source data transferring tool developed by Trendyol with love :orange_heart:","language":"C#","stars":60,"topics":["database","couchbase","sqlserver"],"license":null,"category":"databases-storage","readme_excerpt":"About The Project Transporter is an open-source data transferring tool developed by Trendyol with love . The Why? We developed this tool because we needed to archive our high throughput tables. In other words; we need to move our table data from one database to another. Some of our tables have a daily throughput of more than 1 million. We use these tables constantly. To achieve high performance, we need to archive the data we do not need actively. Some data is stale after a month, some after 6 months. There are a lot of tools out there that do just that, but they are not very flexible or extensible. So we developed our own tool, which has proven to be very useful, and we decided to share it with the community. The How? Transporter has two important concepts; Source and Target. As inferred by names our goal is to move data from Source to Target. Each transporting operation corresponds to a json object. Using Quartz from each json object a cron job is produced. ​ There are two jobs that run in parallel; Polling Job and Transfer Job. Polling Job Steps 1. Select IDs from source table according to given conditions 2. Write selected IDs to interim table Transfer Job Steps 1. Select IDs from the interim table. 2. Select data from the source table that corresponds to selected IDs. 3. Write data to the target. 4. Delete transferred data from the source table. 5. Delete transferred data from the interim respectively. In database processes, data safety is our first concern. Since we del","default_branch":null,"files":null,"tree":[],"storefront":"/r/Trendyol","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/Trendyol/Transporter/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}