{"repo":"sparkdq-community/sparkdq","free":true,"listed":false,"github":"https://github.com/sparkdq-community/sparkdq","clone":"git clone https://github.com/sparkdq-community/sparkdq.git","description":"A lightweight, declarative PySpark framework for data quality validation — check columns, rows, and entire datasets directly in your Spark pipelines","language":"Python","stars":78,"topics":["data-quality","data-validation","pyspark","data-engineering","pyspark-validation","spark-data-quality","apache-spark","databricks","etl"],"license":"Apache-2.0","category":"data-pipelines","readme_excerpt":"SparkDQ — Data Quality Validation for Apache Spark SparkDQ is a lightweight data quality framework built natively for PySpark. You describe what valid data looks like — declaratively via YAML/JSON or through a type-safe Python API — and it validates your DataFrame at row and aggregate level in a single pass. Its defining trait is what it leaves out. SparkDQ is intentionally small in scope and low in complexity — a focused set of checks and a single-pass engine, with no metadata store, orchestration layer, or profiling engine to operate. For most pipelines, that is exactly enough, and the reduced complexity is a feature in itself. That focus is what sets it apart: no JVM bridge like PyDeequ, no complexity overhead like Great Expectations, and no platform lock-in like Databricks dqx. One dependency. No wrappers. No bloat. Quickstart Declarative — checks are passed as dicts, loaded from anywhere: YAML files, JSON, databases, or APIs: Python-native — full type safety and IDE autocompletion: SparkDQ ships with 30+ built-in checks across null validation, numeric ranges, string patterns, date boundaries, schema enforcement, uniqueness, and referential integrity. 🚀 See the official documentation to learn more. Installation For Local Development / Standalone Clusters Install with PySpark included: For Databricks / Managed Platforms Install without PySpark (runtime provided by platform): The framework supports Python 3.11+ and is fully tested with PySpark 3.5.x. SparkDQ will automatic","default_branch":null,"files":null,"tree":[],"storefront":"/r/sparkdq-community","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/sparkdq-community/sparkdq/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}