Great Expectations

The fundamental building block of a Great Expectations suite. An Expectation is a declarative, human-readable statement about your data that can be validated. It defines what "good data" looks like, not how to check it.

• Examples: expect_column_values_to_be_unique, expect_table_row_count_to_be_between, expect_column_values_to_match_regex.
• Expectations are deterministic and produce a clear pass/fail result.
• They can be created by profiling data, written manually, or suggested by the library.

A collection of Expectations grouped together to define the complete quality contract for a specific data asset (e.g., a database table or a file).

• An Expectation Suite acts as a single, versionable specification for data quality.
• It is typically stored as a JSON file, making it portable and easy to version control.
• Suites are executed as a unit by Checkpoints to validate a batch of data.

The primary entry point and configuration manager for a Great Expectations project. The Data Context holds all configuration, connects to data sources, and knows where Expectation Suites, Checkpoints, and validation results are stored.

• It can be configured via a great_expectations.yml file.
• It manages connections to Datasources (e.g., PostgreSQL, S3, Spark).
• It provides the runtime environment for all validation operations.

A configured operation that orchestrates the validation of a data asset. A Checkpoint bundles an Expectation Suite with a specific data batch and defines actions to take based on the validation result.

• Core Function: Run a suite against a dataset, store the results, and optionally trigger notifications (e.g., Slack, email) or block a pipeline.
• They enable pipeline-gated testing by integrating validation into workflow tools like Airflow, Prefect, or Dagster.
• Checkpoints can be run via CLI, Python API, or triggered by external schedulers.

Automatically generated, human-readable documentation for your data quality. Data Docs render Expectation Suites and validation results into HTML reports, providing a clear audit trail.

• They serve as a single source of truth for data quality, accessible to engineers and stakeholders.
• Docs can be hosted locally or on cloud storage (e.g., S3, GCS) for easy sharing.
• Each validation run updates the docs, creating a historical record of data health.

The structured output produced when a Checkpoint runs an Expectation Suite. A Validation Result contains the pass/fail status of each expectation, summary statistics, and a copy of the batch's metadata.

• Results are stored in a Metrics Store (configurable backend) for historical analysis.
• The structured format enables integration with monitoring dashboards and alerting systems.
• Results power Data Docs, providing the evidence for the documented quality state.

Great Expectations is used to programmatically validate data as it moves through pipelines. Engineers embed Expectation Suites—collections of rules—at key stages (e.g., after ingestion or transformation). A Checkpoint executes these suites, comparing incoming data against defined expectations like schema conformity, value ranges, and row counts. Failures can trigger alerts or halt pipelines, preventing corrupt data from propagating to downstream models and dashboards.

The library automates data profiling to discover statistical properties and infer potential expectations. It generates Data Docs, which are human-readable, HTML-based documentation of your datasets and the expectations defined on them. This serves as a living data contract, clearly communicating the expected shape, content, and quality of data assets to all consumers, including data scientists and analysts.

Applying Test-Driven Development (TDD) principles to data engineering, teams write expectations before building or modifying pipelines. This ensures the pipeline logic is designed to meet explicit quality requirements. Expectations act as unit tests for data, validating individual transformations, and integration tests, ensuring correct data flow between components. This methodology shifts quality left, catching issues early in the development cycle.

In production, Great Expectations functions as a data quality monitoring system. Scheduled checkpoints run validation suites on fresh data, enabling continuous testing. Results are logged and can be integrated with monitoring tools like Datadog or PagerDuty. Teams set data quality gates (e.g., 95% of expectations must pass) and receive alerts for anomalies like sudden volume drops, schema drift, or unexpected null values, facilitating rapid data incident management.

An Expectation Suite is a collection of executable assertions about a dataset. Each expectation is a declarative statement, such as expect_column_values_to_be_between. Suites are defined in JSON and can be created manually, inferred via profiling, or generated from data contracts. They encapsulate both simple schema checks and complex business rule validation, serving as the single source of truth for a dataset's expected behavior.

A Checkpoint is the operational configuration that binds an Expectation Suite to a specific data asset (e.g., a database table) and defines validation actions. It is the executable unit of validation. Data Docs are automatically generated, static HTML sites that render Expectation Suites and validation results. They provide a centralized, shareable view of data health, expectations, and lineage, turning validation from a log file into an interactive data observability portal.