Great Expectations
Great Expectations (GX) is an open-source Python framework for defining, running, and documenting automated data quality checks called Expectations.
Overview
Great Expectations (GX) is an open-source Python library for validating, documenting, and profiling data. Instead of writing ad-hoc assertions scattered through a pipeline, you declare Expectations — statements like "this column should never be null" or "values in this column should be between 0 and 100" — and GX runs them against your data, producing a pass/fail report and human-readable documentation (Data Docs) as a byproduct.
How it works
An Expectation Suite is a collection of Expectations tied to a data asset (a table, a file, a query result). GX executes the suite through a Validator, which pushes the checks down to whatever backend holds the data — a pandas DataFrame, a Spark DataFrame, or a SQL engine reached through SQLAlchemy. Results come back as a structured JSON ValidationResult, which can gate a pipeline (fail a dbt run or Airflow DAG on bad data) or feed a dashboard of data quality over time.
Copy code
import great_expectations as gx
context = gx.get_context()
validator = context.sources.add_pandas("orders_source").read_csv("orders.csv")
validator.expect_column_values_to_not_be_null("order_id")
validator.expect_column_values_to_be_between("order_total", min_value=0, max_value=100000)
results = validator.validate()
print(results.success)
GX and DuckDB
Because DuckDB has a SQLAlchemy dialect (duckdb_engine), GX can connect to a DuckDB file — or a MotherDuck database over the same connection string — as a SQL data source, and run Expectations directly against tables with SQL pushed to DuckDB's execution engine rather than pulling everything into pandas first. This is a common pattern for lightweight, local data quality checks in a dbt-DuckDB project: validate a table right after it's built, without provisioning a separate warehouse just to run tests. Support for DuckDB as a SQLAlchemy backend has matured over time, so it's worth checking the current GX version's documentation for any dialect-specific caveats before relying on it in production.
Related terms
A data contract is a formal, agreed-upon specification between a data producer and its consumers that defines the schema, format, semantics, and guarantees (like update frequency or backward-compatibility rules) of a dataset.
Data quality →Data quality is the degree to which data is accurate, complete, consistent, timely, and fit for the purposes it's used for—dashboards, models, and operational decisions.
SQLAlchemy →SQLAlchemy is a popular Python library that provides a flexible way to interact with databases without writing raw SQL code.
DataFrame →A DataFrame is a two-dimensional data structure that organizes data into rows and columns, similar to a spreadsheet or database table.
Data profiling →Data profiling is the process of examining a dataset to understand its structure, content, and quality — things like data types, value distributions, null rates, and cardinality — before using it for analysis or building pipelines on top of it.
Apache Airflow →Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring workflows, where pipelines are defined as directed acyclic graphs (DAGs) in Python.
FAQS
Yes. The core Great Expectations Python library is open source and free to use. GX Cloud, a hosted companion product, is a separate commercial offering.
Yes, via DuckDB's SQLAlchemy dialect (duckdb_engine), GX can connect to a local DuckDB file or a MotherDuck database and run Expectations as SQL pushed down to the engine.
dbt tests are lightweight SQL assertions tied to a dbt project's models. Great Expectations is a standalone, more expressive validation framework that works outside dbt too, with richer Expectation types and built-in documentation and profiling.
