Expectation Suite

Expectations are expressed as human-readable, declarative statements rather than imperative code. Each assertion specifies what the data should look like, not how to check it.

• Example: expect_column_values_to_not_be_null or expect_column_mean_to_be_between
• Decouples validation logic from execution engine
• Enables non-engineers to author and audit data quality rules
• Stored as version-controlled JSON or YAML configurations

Expectation Suites can be auto-generated by profiling a batch of reference data, dramatically accelerating the creation of baseline quality checks.

• Profiler scans data and suggests expectations based on observed statistics
• Captures column types, value ranges, null percentages, and cardinality
• Produces a draft suite that data engineers can refine and customize
• Transforms tribal knowledge about data quality into codified, testable artifacts

Every validation run produces human-readable Data Docs — static HTML documentation that renders expectation results as clean, shareable reports.

• Each expectation displays a pass/fail status with observed values
• Failed expectations highlight the unexpected count and sample failures
• Serves as a living data catalog and quality dashboard
• Can be published to internal portals for cross-team transparency

A Checkpoint bundles an Expectation Suite with a specific data source and action list, creating a repeatable validation workflow.

• Actions include updating Data Docs, sending Slack alerts, or halting pipelines
• Enables CI/CD-style gating: prevent bad data from reaching downstream systems
• Supports batch and streaming validation contexts
• Checkpoints can be triggered via API, CLI, or orchestration tools like Airflow

Suites can validate individual columns, pairs of columns, or entire tables using a taxonomy of expectation types.

• Column Aggregate: expect_column_distinct_values_to_equal_set
• Column Map: expect_column_values_to_match_regex
• Multi-Column: expect_column_pair_values_to_be_equal
• Table-Level: expect_table_row_count_to_be_between
• Composable rules capture complex business logic without custom code

Expectations can be scoped to subsets of data using row_condition parameters, enabling context-sensitive validation.

• Validate only active records: row_condition='status == "active"'
• Apply different thresholds to different customer tiers
• Combine with mostly parameter to tolerate small percentages of anomalies
• Example: expect_column_values_to_be_in_set with mostly=0.95 allows 5% deviation

The execution environment that applies hard-coded logical conditions to data, guaranteeing identical output for identical input. Unlike probabilistic systems, a deterministic engine evaluates expectations as binary pass/fail assertions.

• Executes expect_column_values_to_not_be_null without variation
• Contrasts with ML-based anomaly detection
• Provides auditable, repeatable validation runs

The process of ensuring a dataset strictly conforms to a predefined blueprint of allowed fields, data types, and hierarchical relationships. An Expectation Suite typically includes schema-level expectations as its first line of defense.

• Validates column presence: expect_table_columns_to_match_ordered_list
• Enforces data types: expect_column_values_to_be_of_type
• Catches structural drift before semantic validation begins

A filtering mechanism that accepts or rejects probabilistic model predictions based on whether their associated probability score exceeds a predefined minimum. In the context of validation, this bridges the gap between fuzzy ML outputs and the binary world of expectation suites.

• Rejects low-confidence entity extractions before rule evaluation
• Typical clinical threshold: >0.85 for automated acceptance
• Routes borderline cases to human-in-the-loop review

A formal agreement between data producers and consumers that defines the schema, semantics, and quality guarantees of supplied data. An Expectation Suite serves as the programmatic enforcement layer of a data contract.

• Encodes SLAs as executable expectations
• Prevents silent schema breakage between pipelines
• Shifts quality enforcement left to ingestion time

The automated identification of data points that deviate significantly from a historical baseline or expected distribution. While expectation suites handle known invariants, anomaly detection catches the unknown unknowns.

• Complements static rules with dynamic baselines
• Uses statistical profiling: mean, standard deviation, quantiles
• Flags novel data quality issues for expectation creation

A validation step that verifies the origin, ownership, and transformation history of a data element. Modern expectation suites extend beyond content validation to assert lineage integrity.

• Validates source system identifiers
• Ensures no unauthorized intermediate transformations
• Critical for regulatory compliance and audit readiness