Apache Atlas

Apache Atlas provides hooks and bridges to automatically capture metadata from a wide range of data processing tools and storage systems. This includes:

• Hadoop ecosystem tools: Hive, HBase, Sqoop, Kafka, Spark.
• External systems: Databases (via JDBC), BI tools, and ETL platforms.
• Streaming ingestion: Real-time metadata capture from Apache Kafka topics. This automation creates a live, up-to-date inventory of data assets, their schemas, and their locations without manual intervention.

The platform enables the application of business metadata to technical assets through a flexible classification system. Users can:

• Define classification types like PII, Confidential, or GDPR_Protected.
• Apply tags to tables, columns, or files to denote sensitivity, data domain, or quality.
• Use propagation rules so tags applied at a database level automatically apply to all contained tables. This forms the foundation for policy-based security and access control.

Atlas automatically builds and visualizes data lineage graphs by tracking operations across integrated systems. This shows:

• Origin: Where a dataset or column originated.
• Transformations: Every process (e.g., a Spark job or Hive query) that altered the data.
• Consumption: All downstream reports, models, or applications that depend on the data. This is critical for impact analysis (assessing effects of schema changes) and root-cause analysis for data quality issues.

Beyond technical metadata, Atlas hosts a business glossary to define key organizational terms, metrics, and policies. Features include:

• Term definitions: Authoritative descriptions of business concepts like 'Monthly Active User' or 'Customer Lifetime Value'.
• Semantic linking: Associating glossary terms with physical data assets (e.g., linking the term 'Revenue' to specific database columns).
• Stewardship: Assigning data stewards as owners for terms and assets to ensure accountability and governance.

Atlas integrates with Apache Ranger to enable attribute-based access control (ABAC). Security policies are defined using metadata attributes (tags, classifications) and dynamically enforced. For example:

• A policy can state: "Users tagged as Analyst_EU can only access data tagged Region=EU."
• Masking policies can be triggered based on data classification (e.g., automatically mask SSN columns for non-HR roles).
• Audit trails log all access and policy evaluation events for compliance reporting.

Apache Atlas provides native hooks and listeners for core Hadoop ecosystem components, enabling automatic metadata capture. Key integrations include:

• Apache Hive: Captures table, column, and partition metadata, plus lineage for CREATE TABLE AS SELECT and INSERT queries.
• Apache HBase: Tracks namespace, table, and column family metadata.
• Apache Sqoop: Captures lineage for data movement jobs between relational databases and HDFS/Hive.
• Apache Kafka: Ingests metadata for Kafka topics via the Atlas Kafka Bridge, enabling tracking of streaming data sources. These integrations allow Atlas to build a live map of data assets and their relationships within a data lake.

Atlas extends governance to modern cloud platforms via community-developed connectors and REST APIs. Supported platforms include:

• Snowflake: Metadata for databases, schemas, tables, and views can be ingested using custom scripts or tools like the Atlas Sqoop Hook for Snowflake.
• Amazon Redshift & AWS Glue: Integration is achieved through the AWS Glue Catalog Bridge, synchronizing table definitions and basic lineage.
• Google BigQuery & Azure Synapse: Metadata ingestion typically relies on custom scripts that leverage the Atlas REST API v2 to create entities and relationships, mapping cloud-native objects to Atlas types. This bridges traditional Hadoop governance with cloud-native data platforms.

To capture end-to-end lineage, Atlas integrates with data processing frameworks:

• Apache Spark: The Atlas Spark Hook captures lineage for Spark SQL and DataFrame operations, tracking inputs and outputs to Hive tables or HDFS files.
• Apache NiFi: Uses the Atlas Reporting Task to send flow metadata (processors, queues, provenance events) to Atlas, creating lineage for data flows.
• Apache Airflow: Lineage can be pushed to Atlas via operators or by using the REST API upon task completion, linking DAGs to affected data assets.
• dbt (Data Build Tool): Custom lineage generators can parse dbt manifest files to push model dependencies and transformations into Atlas as lineage graphs.

Atlas acts as the metadata backbone for enterprise governance stacks, integrating with:

• Apache Ranger: The primary security integration. Atlas classifies data assets (e.g., tags a column as PII). Ranger consumes these tags via the Ranger-Atlas Plugin to dynamically enforce access policies (e.g., "Only analysts can query PII data").
• Open Policy Agent (OPA): Classification tags from Atlas can be evaluated as context in OPA policies for fine-grained authorization beyond the Hadoop ecosystem.
• Collibra & Alation: These commercial data catalogs can synchronize with Atlas via its REST API, allowing Atlas to serve as the technical metadata source while business glossary and stewardship are managed elsewhere.

Atlas's extensibility is powered by its well-defined interfaces:

• REST API v2: The core programmatic interface for creating, reading, searching (DSL and Full-Text), and updating metadata entities and lineage. All external tools integrate via this API.
• Type System: All metadata in Atlas is modeled using a flexible type system (Entity, Classification, Relationship types). Integrations must define their own types or extend existing ones (e.g., aws_s3_table, snowflake_view).
• Kafka-based Notification Hook: Atlas publishes all metadata change events (entity create/update/delete) to a Kafka topic. This allows downstream systems (e.g., custom audit logs, search indices) to react to metadata changes in real-time via a event-driven architecture.

Atlas itself can be monitored, and its metadata fuels broader data observability:

• Prometheus & Grafana: Atlas exposes JVM and application metrics (like API latency, active notifications) via a Prometheus endpoint for dashboarding and alerting.
• DataHub & OpenMetadata: While these are alternative catalogs, they can coexist with Atlas. Some organizations use Atlas for deep Hadoop/Spark lineage and policy enforcement, while using DataHub's UI for discovery. Synchronization requires custom scripts.
• Data Quality Tools: Tools like Great Expectations or Monte Carlo can read Atlas lineage to understand data dependencies for root-cause analysis when a quality check fails, identifying upstream sources of corrupted data.