Descriptive Statistics

These statistics identify the central or typical value in a dataset, representing where most data points cluster.

• Mean: The arithmetic average, calculated by summing all values and dividing by the count. Sensitive to extreme outliers.
• Median: The middle value when data is sorted. A robust measure that is not skewed by outliers.
• Mode: The most frequently occurring value in a dataset. Useful for categorical data.

Example: In a dataset of salaries [50k, 55k, 60k, 60k, 250k], the mean is $95k (skewed by the outlier), the median is $60k, and the mode is $60k.

These statistics quantify the variability, spread, or uncertainty within a dataset, indicating how much the values differ from the central tendency.

• Range: The difference between the maximum and minimum values. A simple but volatile measure.
• Variance: The average of the squared differences from the mean. Measures overall spread in squared units.
• Standard Deviation: The square root of the variance. Expressed in the original data's units, it indicates typical deviation from the mean.
• Interquartile Range (IQR): The range between the 25th percentile (Q1) and 75th percentile (Q3). Represents the spread of the middle 50% of data, robust to outliers.

Example: A low standard deviation indicates data points are clustered tightly around the mean, suggesting high consistency.

These statistics describe the form of the data's distribution, going beyond central location and spread to characterize its symmetry and tail behavior.

• Skewness: Measures the asymmetry of a distribution. Positive skew indicates a long tail to the right (mean > median). Negative skew indicates a long tail to the left (mean < median).
• Kurtosis: Measures the 'tailedness' or peakiness of a distribution relative to a normal distribution. High kurtosis indicates heavy tails and a sharp peak (more outliers). Low kurtosis indicates light tails and a flat peak.

Example: In financial returns, positive skew is desirable (chance of large gains), while high kurtosis ('fat tails') indicates higher risk of extreme market moves.

These statistics divide a sorted dataset into equal-sized subgroups, describing the relative standing of a particular value within the overall distribution.

• Percentiles: Values below which a given percentage of observations fall. The 50th percentile is the median.
• Quartiles: Specific percentiles that divide data into four quarters: Q1 (25th), Q2 (50th/median), Q3 (75th).
• Five-Number Summary: A concise description consisting of the minimum, Q1, median, Q3, and maximum. The basis for a box plot visualization.

Example: If a test score is at the 90th percentile, it is higher than 90% of all scores. The IQR (Q3 - Q1) is a core measure for anomaly detection.

These are the most fundamental descriptive measures, providing counts and proportions that form the basis for understanding data composition.

• Count / Frequency: The absolute number of occurrences for each unique value or category.
• Relative Frequency / Proportion: The count divided by the total number of observations, often expressed as a percentage.
• Cumulative Frequency: The running total of frequencies, showing the number of observations up to a certain value.

Example: Profiling a country column would involve calculating the frequency of each country code. These counts are essential for calculating the completeness metric (non-null count) and assessing cardinality.

These statistics summarize the strength and direction of a linear relationship between two numeric variables, a key part of cross-dataset analysis and data relationship mapping.

• Covariance: Indicates the direction of the linear relationship. Positive covariance implies variables tend to move together.
• Correlation Coefficient (Pearson's r): A normalized measure of linear association ranging from -1 (perfect negative correlation) to +1 (perfect positive correlation). A value of 0 indicates no linear relationship.

Example: In data profiling, calculating correlation between columns can reveal unexpected relationships, potential redundancy, or aid in functional dependency discovery. It does not imply causation.

Descriptive statistics are calculated during initial data profiling to create a quantitative snapshot of a healthy dataset. This snapshot includes:

• Central Tendency: Mean, median, and mode for numeric columns.
• Dispersion: Standard deviation, variance, and range.
• Distribution Shape: Skewness and kurtosis. These metrics become the statistical signature or baseline. In observability platforms, current statistics are continuously compared against this baseline to flag data drift—unexpected changes in the underlying data distribution that can degrade model performance.

Statistics like the interquartile range (IQR) and Z-scores are used to programmatically identify outliers. For example, a rule might flag any value more than 3 standard deviations from the mean. In observability, these calculations run over sliding windows on streaming data. A sudden spike in the count of nulls (completeness metric) or a shift in the median value triggers alerts, enabling teams to investigate pipeline breaks, sensor failures, or fraudulent activity before downstream impacts occur.

Beyond single metrics, descriptive statistics track the health of entire value distributions. Observability tools use:

• Histograms to monitor the shape of categorical and binned numeric data.
• Cardinality (count of distinct values) to detect unexpected explosions or collapses, which may indicate a broken join or encoding error.
• Quantiles (e.g., 5th, 50th, 95th percentiles) to understand boundaries and track their movement over time. A change in the 95th percentile for a transaction amount could signal new user behavior or a data ingestion error.

Descriptive statistics are the engine for core data quality dimensions:

• Completeness: Calculated as (Non-Null Count) / (Total Count).
• Uniqueness: Cardinality analysis determines if a column is a potential primary key.
• Consistency: Statistical checks on value formats (e.g., all strings in a column match a regex pattern for email).
• Freshness: While temporal, it relies on statistics like MAX(timestamp) to measure latency. These quantified metrics are aggregated into data health scores and tracked via Service Level Objectives (SLOs) for data reliability engineering.

Statistics inform infrastructure and performance optimization:

• Data Volume Analysis: Row counts and byte sizes track dataset growth for capacity planning.
• Sparsity Analysis: The percentage of zeros or nulls helps choose efficient storage formats (e.g., Parquet for sparse data).
• Temporal Analysis: Profiling statistics over time reveals seasonal patterns or growth trends, allowing for predictive scaling of compute resources. Observability platforms use these profiles to alert on unexpected volume drops (a broken source) or surges (a potential DDoS attack log).

Descriptive statistics define the assertions for automated data tests. Engineers codify expectations about their data using statistical rules:

• assert column_mean between (10, 20)
• assert column_null_rate < 0.01
• assert column_stddev < 5 These tests run after each pipeline execution or on a schedule. This practice, known as Statistical Process Control for Data, moves data quality checks left in the development cycle. Failed statistical assertions create incidents that tie directly to the affected data lineage, speeding up root cause analysis.

Measures that identify the central or typical value in a dataset.

• Mean: The arithmetic average of all values.
• Median: The middle value when data is sorted; robust to outliers.
• Mode: The most frequently occurring value. These metrics answer the question: 'Where is the data centered?'

Metrics that quantify the spread or variability of data points around the central tendency.

• Range: The difference between the maximum and minimum values.
• Variance: The average of the squared differences from the mean.
• Standard Deviation: The square root of the variance, expressed in the original data's units.
• Interquartile Range (IQR): The range between the 25th and 75th percentiles, describing the spread of the middle 50% of data.

Statistics that describe the symmetry and peakedness of a data distribution.

• Skewness: Measures the asymmetry of the distribution. Positive skew indicates a long tail to the right; negative skew indicates a long tail to the left.
• Kurtosis: Measures the 'tailedness' and peakedness. High kurtosis indicates heavy tails and a sharp peak; low kurtosis indicates light tails and a flat distribution. These are critical for understanding the underlying data generation process.

Values that divide a dataset into equal-sized subgroups.

• Quartiles: Divide data into four parts (Q1=25th percentile, Q2=50th percentile/median, Q3=75th percentile).
• Percentiles: Divide data into 100 parts. The 95th percentile is the value below which 95% of observations fall. These are essential for understanding data ranges, detecting outliers (e.g., using the 1.5*IQR rule), and setting data quality thresholds.

A summary of how often each distinct value or range of values occurs in a dataset.

• Histogram: A bar chart visualizing the frequency distribution of a continuous variable.
• Frequency Table: A tabular summary showing counts and percentages for categorical or binned numerical data. This analysis reveals patterns, common values, and potential data entry errors.

A measure of the strength and direction of a linear relationship between two continuous variables.

• Pearson Correlation Coefficient (r): Ranges from -1 (perfect negative correlation) to +1 (perfect positive correlation). A value of 0 indicates no linear correlation.
• Covariance: Indicates the direction of the linear relationship but is not normalized, making its magnitude difficult to interpret. Correlation is a foundational step for feature selection and understanding multivariate relationships.