Apache Parquet

Unlike row-based formats (e.g., CSV, Avro), Parquet stores data column-by-column. This fundamental design choice provides significant performance advantages for analytical queries that typically scan specific columns rather than entire rows.

• I/O Efficiency: Queries reading only a few columns skip reading entire rows from disk, drastically reducing I/O.
• Data Locality: Values from the same column are stored contiguously, enabling highly efficient compression and vectorized processing.
• Predicate Pushdown: Query engines can skip reading entire row groups by evaluating filters directly on column metadata.

Parquet employs sophisticated, type-aware encoding schemes followed by general-purpose compression, achieving high compression ratios and fast decoding.

• Encoding Schemes: Techniques like dictionary encoding (for low-cardinality columns), run-length encoding (RLE) (for repeated values), and delta encoding exploit data patterns within a column.
• Two-Stage Process: Data is first encoded (transformed for efficiency), then compressed using algorithms like Snappy, GZIP, or LZ4. This order maximizes compression effectiveness.
• Column-Specific Optimization: Each column can use the encoding and compression codec best suited to its data type and distribution.

Parquet natively supports complex, hierarchical data structures through its repetition and definition level encoding (inspired by Google's Dremel paper). This is crucial for semi-structured data like JSON.

• Schema Representation: Nested fields (structs, lists, maps) are flattened into a columnar format while preserving the original structure's semantics.
• Efficient Storage: Null values and repeated fields in nested records are stored without redundancy.
• Compatibility: Provides a lossless, efficient serialization format for dataframes (Pandas, Spark) and protocol buffers/Avro schemas.

Parquet files embed rich metadata—including column-level statistics—enabling query engines to skip irrelevant data during scans, a process known as predicate pushdown.

• File & Row Group Metadata: Each file contains a footer with schema, and each row group stores per-column min/max values, null counts, and distinct counts.
• Data Skipping: A query engine can determine that a row group's min/max range does not satisfy a filter (e.g., date > '2024-01-01') and skip reading it entirely.
• Bloom Filters: Optional Bloom filters provide probabilistic data structures for efficient membership tests on specific column values, further accelerating equality predicates.

Parquet supports safe, backward- and forward-compatible schema changes, allowing data producers and consumers to evolve independently—a critical feature for long-lived data lakes.

• Compatibility Rules: Columns can be added (new readers see them, old readers ignore them) or removed (old readers see them as null, new readers ignore them). Column types can be promoted (e.g., int -> long).
• Metadata Handling: Schema information is stored in the file footer. Readers use their own projected schema to read a subset of columns, gracefully handling missing fields.
• Use Case: Enables practices like schema-on-read, where data is written with one schema and later interpreted with another for different analytical purposes.

Parquet is a de facto standard for analytical data storage, with deep, native integration across the entire data processing ecosystem.

• Query Engines: Apache Spark, Presto/Trino, Apache Hive, DuckDB, and BigQuery (externally) all have optimized Parquet readers.
• DataFrames: Libraries like Pandas (via PyArrow), Polars, and Apache Arrow provide first-class read/write support.
• Cloud Object Stores: Optimized for reading from systems like Amazon S3, Azure Data Lake Storage, and Google Cloud Storage, often with parallelized reads.
• Serialization Frameworks: Can serve as an efficient storage layer for data defined by Avro, Protocol Buffers, or Thrift schemas.

Parquet is increasingly used to store high-volume telemetry data, such as application logs, metrics, and distributed traces, in cost-effective data lakes. Instead of expensive indexed storage in tools like Elasticsearch for all historical data, a common pattern is:

1. Hot Data: Recent logs are indexed for real-time search.
2. Cold/Warm Data: Older logs are aggregated and written to Parquet files in object storage. Analytical queries on historical logs (e.g., trend analysis, forensics) are then run using SQL engines on the Parquet archive. This provides deep analytical capability at a fraction of the cost of maintaining all data in an indexed system.

Columnar storage is a data organization method where values from each table column are stored contiguously on disk, as opposed to row-oriented storage which stores all values for a single record together. This is the foundational principle behind Parquet's efficiency.

• Analytical Performance: For queries that aggregate over specific columns (e.g., SUM(sales), AVG(temperature)), a columnar format reads only the necessary column data, dramatically reducing I/O.
• Compression Efficiency: Data within a single column is typically homogeneous (e.g., all integers), allowing for highly effective compression algorithms like run-length encoding and dictionary encoding, which Parquet implements.
• Contrast with Row Stores: Traditional OLTP databases (e.g., PostgreSQL, MySQL) use row-oriented storage, which is optimized for transactional workloads where entire records are inserted or retrieved at once.

Predicate pushdown is a critical query optimization technique where filtering conditions (predicates) are pushed down to the data storage layer, minimizing the amount of data that must be read and transferred. Parquet's design enables this.

• How Parquet Enables It: Parquet files contain column statistics and min/max values for data chunks within each column (stored in the footer). A query engine can read these statistics first and skip entire row groups if the filter condition falls outside the recorded range.
• Performance Impact: For a query like SELECT * FROM logs WHERE date = '2024-01-01', the engine can use the statistics in the date column's metadata to read only the row groups containing that date, potentially skipping 99% of the file.
• Essential Feature: This is a primary reason columnar formats like Parquet are indispensable for big data analytics, turning full-table scans into highly selective reads.

Delta Lake, Apache Iceberg, and Apache Hudi are table format projects that provide ACID transactions, time travel, and metadata management on top of cloud object stores. They are not file formats themselves but use Parquet as their primary underlying data file format.

• The Layer Above: These formats add a transactional metadata layer (stored as JSON or other formats) that manages a collection of Parquet files as a single, reliable table. They solve the "many-files-in-a-directory" problem of raw data lakes.
• Core Features: They enable ACID guarantees, snapshot isolation, upserts/merges, and rollbacks—operations that are not natively possible with plain Parquet files.
• Parquet's Role: The actual columnar data is stored in standard Parquet files. The table format's metadata tracks which Parquet files are part of the current table version, enabling efficient time-travel queries by pointing to different sets of files.