{"repo":"GoogleCloudPlatform/market-data-transcoder","free":true,"listed":false,"github":"https://github.com/GoogleCloudPlatform/market-data-transcoder","clone":"git clone https://github.com/GoogleCloudPlatform/market-data-transcoder.git","description":"ffmpeg for market data","language":"Python","stars":46,"topics":["automation","binary","binaryencoding","devops","exchanges","fix","fixprotocol","itch","marketdata","sbe"],"license":"Apache-2.0","category":"deployment-docker-iac","readme_excerpt":"Ingest high-performance exchange feeds into Google Cloud This is not an official Google product or service Introduction The Datacast transcoder is a schema-driven, message-oriented utility to simplify the lossless ingestion of common high-performance electronic trading data formats to Google Cloud. Electronic trading venues have specialized data representation and distribution needs. In particular, efficient message representation is a high priority due to the massive volume of transactions a venue processes. Cloud-native APIs often use JSON for message payloads, but the extra bytes required to represent messages using high-context encodings have cost implications in metered computing environments. Unlike JSON, YAML, or even CSV, binary-encoded data is low-context and not self-describing -- the instructions for interpreting binary messages must be explicitly provided by producers separately and in advance, and followed by interpreters. The architecture of the transcoder relies on several principal abstractions, detailed below: Schema A schema (also known as a data dictionary) is similar to an API specification, but instead of describing API endpoint contracts, it describes the representative format of binary messages that flow between systems. The closest comparison might be drawn with table definitions supported by SQL Data Definition Language, but these schemas are used for data in-motion as well as data at-rest. The transcoder's current input schema support is for Simple B","default_branch":null,"files":null,"tree":[],"storefront":"/r/GoogleCloudPlatform","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/GoogleCloudPlatform/market-data-transcoder/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}