{"repo":"ClickHouse/FastFormats","free":true,"listed":false,"github":"https://github.com/ClickHouse/FastFormats","clone":"git clone https://github.com/ClickHouse/FastFormats.git","description":"A benchmark for data formats","language":"HTML","stars":16,"topics":["analytics","arrow","avro","benchmark","bson","capnproto","clickhouse","cpp","csv","database"],"license":null,"category":"databases-storage","readme_excerpt":"FastFormats: A Benchmark for ClickHouse Input Formats Overview FastFormats is a benchmark designed to compare the ingestion performance of different data formats in ClickHouse. It measures server-side insert speed , CPU and memory efficiency , and the impact of compression, pre-sorting, and batching on ingestion performance. ClickHouse supports 70+ input formats, but choosing the right one can significantly impact performance. FastFormats systematically evaluates these formats to identify the most efficient options for high-throughput ingestion. Dataset FastFormats uses the real-world web analytics dataset from ClickBench. Benchmark Scope FastFormats evaluates: - Insert speed : How quickly the server processes data in different formats. - Resource efficiency : CPU and memory usage per insert. - Compression impact : How different compression methods affect insert performance. - Pre-sorting benefits : Whether sorting data before sending improves ingestion speed. - Batching efficiency : The effect of varying batch sizes on performance. The benchmark includes 60+ formats , covering: - Binary columnar formats (e.g., Native, Parquet, Arrow) - Binary row-oriented formats (e.g., RowBinary, Avro, Protobuf) - Text-based formats (e.g., CSV, TSV, JSONEachRow) How It Works FastFormats runs a nested loop over multiple test dimensions: - Batch sizes : 10k, 100k, and 1M rows per insert. - Formats : CSV, TSV, JSONEachRow, Native, Parquet, Arrow, and more. - Pre-sorting : Yes/No. - Compression","default_branch":null,"files":null,"tree":[],"storefront":"/r/ClickHouse","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/ClickHouse/FastFormats/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}