Anomaly Detection

The foundational process of establishing a normal operating profile for web traffic. This involves calculating the mean, median, and standard deviation of metrics like requests per second, unique URLs accessed, and session duration over a defined training window. Once the baseline is established, any observation falling outside a configurable threshold—typically three standard deviations from the mean—is flagged as anomalous. This technique is highly effective against high-volume scrapers that deviate from human browsing cadence.

Detects bots by monitoring the inter-arrival time between consecutive HTTP requests. Human users exhibit variable, stochastic timing patterns with natural pauses for reading and interaction. Automated scripts typically generate requests at machine-gun speeds with unnaturally consistent intervals. A Poisson distribution is often used to model human request rates; traffic that violates this distribution with sub-second, perfectly periodic requests is a high-confidence indicator of a scripted agent.

Examines the sequence and topology of page transitions within a session. Human users follow organic, referential paths—clicking links, reading content, and navigating with purpose. Scrapers often traverse the site graph in unnatural patterns:

• Depth-first crawling: Recursively following every link on a page before returning.
• Breadth-first crawling: Systematically enumerating all URLs at a directory level.
• Random jumping: Accessing deeply nested pages without intermediate navigation. Markov chain models can score the probability of a given transition sequence, flagging low-probability paths as automated.

Identifies bots by analyzing the distribution of session durations. Human sessions typically follow a log-normal distribution, with most visits lasting a few minutes and a long tail of extended engagements. Scraping sessions often manifest as extreme outliers—either thousands of requests in seconds or persistent, multi-hour connections that never idle. Monitoring the ratio of active interaction events to total session time provides a powerful signal for distinguishing human browsing from automated data extraction.

Measures the randomness and diversity of requested URLs within a session. Human traffic exhibits high entropy in the types of assets requested—a mix of HTML pages, CSS, images, and API calls with varied query parameters. Scrapers targeting structured data often show abnormally low entropy, requesting only product pages or sequentially iterating through numeric IDs. Shannon entropy calculated over the URL path and query string character distributions provides a mathematical basis for flagging highly predictable, automated request patterns.

An unsupervised machine learning algorithm specifically designed for anomaly detection. Unlike methods that profile normal points, Isolation Forest explicitly isolates anomalies by randomly selecting a feature and a split value. Anomalous data points—being few and different—require fewer random partitions to be isolated, resulting in shorter path lengths in the tree ensemble. This technique is computationally efficient, requires no distance calculations, and excels at detecting scraping traffic that forms sparse, distinct clusters in high-dimensional feature space.

The analysis of human interaction patterns to distinguish organic users from automated scripts. Unlike static fingerprinting, behavioral biometrics examines dynamic signals:

• Mouse movement trajectories — curvature, acceleration, and jitter
• Keystroke dynamics — typing rhythm, dwell time, and flight time
• Touch pressure and gesture patterns on mobile devices
• Scroll behavior — speed, pauses, and micro-adjustments

Scripted bots exhibit unnaturally linear or perfectly consistent interaction patterns that deviate sharply from the chaotic, non-linear signatures of human motor control. These deviations serve as high-confidence anomaly signals.

A statistical profiling technique that establishes normal session duration distributions and flags outliers indicative of automated scraping. Key metrics include:

• Time-on-page distributions — scrapers often exhibit sub-second or unnaturally uniform dwell times
• Inter-request intervals — humans show variable delays; bots display clockwork regularity
• Session depth patterns — the number of pages visited per session follows predictable human curves

Deviation from established baselines triggers anomaly scoring. A session that traverses 500 product pages in 30 seconds with precisely 3.6-second intervals between requests represents a clear statistical outlier from legitimate browsing behavior.

The examination of page transition sequences to identify non-human traversal patterns through a web property. Legitimate users follow predictable navigation graphs:

• Referrer chain integrity — humans arrive from search engines, email links, or internal navigation
• Click depth progression — gradual drilling from category to product to detail pages
• Markov chain modeling — transition probabilities between pages form stable distributions

Scrapers often exhibit depth-first crawling or access pages directly via URL lists without referrer headers, creating navigation graphs that violate the expected Markov transition matrix. These structural anomalies in the clickstream are highly indicative of automated extraction.

The statistical modeling of HTTP request velocity to establish per-endpoint, per-session, and per-IP baselines. This goes beyond simple rate limiting by analyzing:

• Burst patterns — humans generate irregular bursts; scrapers sustain flat, maximum-throughput curves
• Diurnal rhythm alignment — legitimate traffic follows timezone-correlated activity cycles
• Endpoint access distribution — scrapers disproportionately hit high-value data endpoints

Anomaly detection systems apply z-score analysis and interquartile range methods to flag request rates exceeding dynamic thresholds. A residential IP issuing 200 requests per minute to a product API at 3 AM local time triggers multiple simultaneous anomaly signals.

A machine learning approach that groups traffic sessions into behavioral clusters without pre-labeled training data, enabling the discovery of novel scraping patterns. Common algorithms include:

• DBSCAN — density-based clustering that identifies sessions with similar feature vectors
• Isolation Forest — isolates anomalous sessions by their susceptibility to random partitioning
• Autoencoders — neural networks that learn to reconstruct normal traffic; high reconstruction error signals anomalies

When a new scraping tool emerges with no known signature, unsupervised methods detect it by identifying sessions that form tight, isolated clusters far from the dense regions of legitimate human traffic in high-dimensional feature space.

A complementary detection mechanism that embeds invisible elements within pages to catch automated scrapers that interact programmatically with the DOM. These traps exploit the fundamental difference between human visual perception and bot parsing:

• Hidden links — CSS-hidden anchor tags invisible to users but present in the HTML tree
• Invisible form fields — input elements positioned off-screen or behind other elements
• Honeytoken data — unique synthetic values injected into responses to trace unauthorized redistribution

A scraper that follows a hidden link or populates an invisible form field generates a definitive anomaly signal. Unlike behavioral analysis, honeypot interactions provide deterministic proof of automation with near-zero false positives.