Expectation Suite

The core building blocks of a suite. Each expectation is a declarative statement about a property the data must have. Common types include:

• Column-level expectations: expect_column_values_to_not_be_null, expect_column_values_to_be_between.
• Table-level expectations: expect_table_row_count_to_equal, expect_table_columns_to_match_ordered_list.
• Cross-column expectations: expect_column_pair_values_to_be_equal, expect_multicolumn_sum_to_equal. These are defined in code (Python) or configuration (JSON/YAML) and produce a pass/fail result with metrics.

An Expectation Suite is bound to a specific data asset, such as a database table, file path, or Spark DataFrame. The batch definition specifies how to fetch a sample of this asset for validation. It includes:

• Datasource name: Connection to the underlying data store.
• Data connector: Logic to access a specific subset (e.g., a partition).
• Data asset identifier: The name of the table, file, or query. This binding ensures the suite's rules are applied to the correct, consistently defined dataset.

When a suite is run, it produces a structured validation result. This is a JSON-like object containing:

• Success status: Overall pass/fail for the suite.
• Expectation results: A list detailing the outcome of each individual expectation.
• Statistics: Counts of evaluated rows, successful expectations, and failures.
• Meta-data: Timestamp, batch identifier, and execution engine info. This standardized schema enables result aggregation, dashboarding, and integration with alerting systems.

Suites contain contextual information beyond the raw rules. This meta-data includes:

• Suite name and version: For tracking changes over time.
• Data context: Links to the broader Great Expectations project.
• Expectation Kwargs: The specific parameters for each rule (e.g., min_value=0, max_value=100).
• Human-readable descriptions: Optional notes explaining the business rationale for an expectation. This documentation turns the suite into a living specification, making data contracts explicit and auditable.

While not stored within the suite itself, an Expectation Suite is designed to be executed by a checkpoint. The checkpoint configuration defines:

• Which suite(s) to run.
• Against which data batch (often dynamically defined at runtime).
• What actions to take on success or failure (e.g., store results, send a Slack alert, update a Data Docs site). The suite provides the 'what' (the rules), and the checkpoint provides the 'when' and 'how' for execution.

A key feature is the automatic generation of Data Docs—human-readable HTML documentation for the suite. Renderers are components that translate the suite's expectations and validation results into visual reports. This includes:

• Expectation suite overviews: Listing all rules.
• Detailed expectation pages: Showing logic, parameters, and example failures.
• Validation result pages: Displaying the outcome of a specific run. This creates a shared, always-updated source of truth for data quality standards across engineering and business teams.

A declarative or programmatic statement that defines a condition data must satisfy to be considered valid. This is the atomic unit of an Expectation Suite.

• Examples: A column must be non-null, values must fall within a defined range, or a set of columns must be unique.
• Implementation: Can be expressed in SQL, Python, or configuration languages like YAML.
• Purpose: Encodes a specific, testable constraint derived from business logic, schema requirements, or statistical norms.

A configured operation that runs an Expectation Suite against a specific dataset and triggers actions based on the validation results. It is the execution engine for a suite.

• Function: Connects a data asset (e.g., a database table) with an Expectation Suite and an action (e.g., send a notification, update a dashboard).
• Automation: Checkpoints are typically triggered by pipeline orchestration tools (like Apache Airflow) or by events (like new data arrival).
• Output: Produces a Validation Result document detailing which expectations passed or failed.

A formal agreement that specifies the expected schema, data types, freshness, and quality guarantees for a data product. An Expectation Suite can serve as the executable, technical manifestation of a data contract.

• Relationship: While a contract is a business agreement between producer and consumer, an Expectation Suite is the enforcement mechanism.
• Scope: Contracts often include SLAs and broader semantics; Expectation Suites focus on executable validation rules.
• Evolution: Changes to an Expectation Suite should reflect negotiated updates to the underlying data contract.

A software methodology applied to data engineering where data quality tests (like Expectation Suites) are written before the pipeline code that produces the data. This ensures the pipeline is built to satisfy explicit quality requirements.

• Workflow: 1. Define expectations for the target dataset. 2. Write pipeline code. 3. Run validation. 4. Iterate until all expectations pass.
• Benefit: Catches logic errors and data quality issues at the earliest possible stage, reducing downstream data incidents.
• Mindset: Shifts data validation from a post-hoc audit to an integral part of the development lifecycle.

A data quality test (or entire Expectation Suite) whose failure is configured to prevent a data pipeline from proceeding to the next stage. It acts as a quality gate to block invalid data from propagating.

• Implementation: A Checkpoint is configured to raise an error or fail a CI/CD job if validation fails.
• Use Case: Critical for enforcing data quality gates before data lands in a production analytics table or ML feature store.
• Trade-off: Balances data safety against pipeline resilience; not all tests should be blocking.