{"repo":"sevagh/pq","free":true,"listed":false,"github":"https://github.com/sevagh/pq","clone":"git clone https://github.com/sevagh/pq.git","description":"a command-line Protobuf parser with Kafka support and JSON output","language":"Rust","stars":175,"topics":["protobuf","parser","kafka"],"license":"MIT","category":"data-pipelines","readme_excerpt":"pq protobuf to json deserializer, written in Rust pq is a tool which deserializes protobuf messages given a set of pre-compiled .fdset files. It can understand varint/leb128-delimited/i32be streams, and it can connect to Kafka. pq will pretty-print when outputting to a tty, but you should pipe it to jq for more fully-featured json handling. Download pq is on crates.io: cargo install pq . You can also download a static binary from the releases page. Usage new You can now pass in a proto file and have pq compile it on the fly using protoc : Use PROTOC and PROTOC INCLUDE to control the executed protoc binary and configure the -I=/proto/path argument (design copied from prost). To set up, put your .fdset files in /.pq , /etc/pq , or a different directory specified with the FDSET PATH env var: You can specify additional fdset directories or files via options: Pipe a single compiled protobuf message: Pipe a varint or leb128 delimited stream: Consume from a Kafka stream: Convert a Kafka stream to varint-delimited: Pipe kafkacat output to it: Compile without kafka To compile pq without kafka support, run: Compile for Windows 1. Install Visual Studio Installer Community edition 2. Run the installer and install Visual Studio Build Tools 2019 . You need the C++ Build Tools workload. Note that you can't just install the minimal package, you also need MSVC C++ x64/86 Build Tools and Windows 10 SDK . 3. Open x64 Native Tools Command Prompt from the start menu. 4. Download and run rustup-in","default_branch":null,"files":null,"tree":[],"storefront":"/r/sevagh","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/sevagh/pq/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}