Data Pipeline

The initial stage where data is collected from diverse source systems and brought into the processing environment. This involves connecting to APIs, databases, message queues, and file systems. Key patterns include:

• Batch Ingestion: Periodic, scheduled data transfers (e.g., nightly ETL jobs).
• Stream Ingestion: Continuous, real-time data collection from event streams (e.g., using Apache Kafka or Amazon Kinesis).
• Change Data Capture (CDC): Capturing only incremental changes from source systems to propagate updates efficiently. The reliability and latency of this stage set the foundation for all downstream processing.

The stage where raw data is cleansed, enriched, and structured into a usable format. This involves applying business logic and quality rules. Core operations include:

• Data Validation: Enforcing schema rules, data types, and value constraints.
• Data Cleansing: Handling missing values, correcting errors, and standardizing formats.
• Data Enrichment: Joining datasets or augmenting records with external information.
• Aggregation: Summarizing data (e.g., calculating daily totals). Processing can occur in batch frameworks like Apache Spark or streaming engines like Apache Flink.

The component responsible for persisting processed data and making it available to consumers. The choice of storage layer depends on the access pattern and data structure.

• Data Warehouses (e.g., Snowflake, BigQuery): Optimized for complex analytical queries on structured data.
• Data Lakes (e.g., Amazon S3, ADLS): Store vast amounts of raw and processed data in open formats (Parquet, Avro).
• Data Lakehouses (e.g., Databricks Delta Lake): Combine lake storage with warehouse management features like ACID transactions.
• OLTP Databases & Caches: Serve low-latency requests for applications (e.g., PostgreSQL, Redis).

The central nervous system that schedules, coordinates, and monitors the execution of pipeline tasks. Orchestrators manage dependencies, handle failures, and ensure tasks run in the correct order and at the right time.

• Key Functions: Task scheduling, dependency management, error handling, retry logic, and alerting.
• Common Tools: Apache Airflow, Dagster, Prefect, and cloud-native services like AWS Step Functions and Google Cloud Composer. Orchestration is critical for turning a collection of scripts into a reliable, maintainable production system.

The integrated systems for monitoring, validating, and ensuring the health of data throughout the pipeline. This component is proactive, detecting issues before they impact downstream consumers.

• Data Quality Checks: Programmatic validation of freshness, volume, schema, and custom business rules.
• Anomaly Detection: Identifying statistical drifts or unexpected patterns in data distributions.
• Lineage Tracking: Mapping data flow from source to destination for impact analysis and debugging.
• Monitoring & Alerting: Dashboards and notifications for pipeline health, data SLAs, and failures.

The systematic handling of data about the data pipeline itself. This includes technical metadata (schemas, lineage), operational metadata (run times, logs), and business metadata (data definitions, owners).

• Metadata Catalogs (e.g., DataHub, OpenMetadata): Serve as a centralized inventory for discovering, understanding, and governing data assets.
• Schema Registries (e.g., Confluent Schema Registry): Manage and enforce schema evolution for data in motion (e.g., Apache Kafka topics). Effective metadata management enables data discovery, governance, and trust, turning raw data into a managed asset.

The tracking of data's origin, movement, transformations, and dependencies across systems and processes over its lifecycle. It provides a map for:

• Impact Analysis: Understanding which downstream reports or models will be affected by a change in a source table.
• Root Cause Debugging: Tracing an erroneous output in a dashboard back to the specific transformation step where the error was introduced.
• Compliance Auditing: Demonstrating the provenance and handling of regulated data (e.g., PII, financial records). Modern tools automatically infer lineage by parsing SQL queries, pipeline code (e.g., Airflow DAGs, dbt models), and job execution logs.

A formal, versioned agreement between a data producer (e.g., a service team) and data consumers (e.g., analytics, ML teams) that specifies the programmatic interface for a data product. It codifies expectations to prevent pipeline breaks and includes:

• Schema: The exact structure, data types, and allowed values (enums).
• Semantics: The business meaning of fields and allowable transformations.
• Service Level Objectives (SLOs): Guarantees for freshness (e.g., data updated every hour), latency, and availability.
• Evolution Rules: Policies for backward-compatible changes (e.g., only adding nullable columns). Contracts are enforced via automated testing in CI/CD pipelines, failing builds if producers generate non-compliant data.

A design pattern that identifies and captures incremental changes (inserts, updates, deletes) made to data in a source database, propagating them to downstream systems. This is a critical technique for building real-time data pipelines and avoiding full-table reloads.

Common Implementation Methods:

• Database Log Scraping: Reading the transaction log (e.g., MySQL binlog, PostgreSQL WAL) to capture changes with low latency.
• Trigger-Based: Using database triggers to write changes to a separate shadow table.
• Query-Based: Polling a table for changes using a last_updated timestamp column (higher latency).

Tools: Debezium (open-source, log-based), AWS DMS, Fivetran, Striim.

The practice of managing changes to a data schema over time while maintaining compatibility with existing consumers. A Schema Registry is a central service that stores and governs these schemas (e.g., Avro, Protobuf, JSON Schema) for data in motion (streaming).

Key Concepts:

• Backward Compatibility: New schema can read data written with the old schema (e.g., adding a new optional field). Consumers can upgrade at their own pace.
• Forward Compatibility: Old schema can read data written with the new schema (e.g., removing an optional field). Allows producers to upgrade first.
• Compatibility Checks: The registry validates new schema submissions against these rules before allowing them in production, preventing breaking changes in Kafka topics or event streams.

The measure of the health and state of data in a pipeline, extending beyond basic monitoring. It applies SRE principles to data systems, using automated checks to answer: "Is my data correct, fresh, and reliable?"

Five Pillars:

• Freshness: Is the data up-to-date? When was the last pipeline run?
• Quality: Does the data conform to expectations (e.g., non-null rates, value distributions, uniqueness)?
• Volume: Has the expected amount of data arrived (unexpected drops or spikes)?
• Schema: Has the structure of the data changed unexpectedly?
• Lineage: As covered above, for impact analysis. Platforms like Monte Carlo, BigEye, and Datafold automate the detection of anomalies across these dimensions.

A reusable, high-quality data asset—such as a curated dataset, a machine learning model feature set, or an API—that is packaged, documented, and managed as a product. This is a core concept in the Data Mesh paradigm.

Key Attributes:

• Owned by a Domain: A specific business unit (e.g., finance, inventory) is responsible for its quality and lifecycle.
• Discoverable: Listed in a central data catalog with clear metadata.
• Addressable: Accessed via a standard, reliable path (e.g., s3://data-products/finance/ledger).
• Trustworthy: Backed by SLOs for quality and freshness, often enforced via data contracts.
• Interoperable: Built on global standards for security, governance, and metadata. This shifts the mindset from centralized pipeline management to decentralized, product-oriented data ownership.