Cosine Distance

Cosine distance is magnitude-agnostic, meaning it ignores the absolute length of vectors and focuses solely on their orientation. This property is critical in text analysis, where a short document and a long document discussing the same topic should be considered similar. Two vectors pointing in the same direction will have a cosine distance of 0, regardless of whether one has a magnitude of 10 and the other 1,000. This is achieved by the inherent normalization in the dot product formula: cos(θ) = (A · B) / (||A|| × ||B||).

The output of the cosine similarity function is strictly confined to the interval [-1, 1], which is then often converted to a distance metric bounded in [0, 2]. A similarity score of 1 indicates perfectly identical orientation (0° angle), 0 indicates orthogonality (90° angle, no correlation), and -1 indicates diametrically opposed vectors (180° angle). This bounded nature makes thresholds highly interpretable: a cosine similarity above 0.8 reliably indicates strong directional alignment, simplifying the definition of 'local' in LIME's neighborhood generation.

Cosine distance is computationally efficient for sparse, high-dimensional data like TF-IDF vectors. The calculation only needs to consider non-zero dimensions shared between two vectors. For a TF-IDF matrix where most terms are absent from any given document, the dot product computation collapses from O(n) to O(min(nnz(A), nnz(B))), where nnz is the number of non-zero entries. This sparsity exploitation is essential for LIME's perturbation sampling, which must rapidly compute thousands of distances between the original text instance and its masked variations.

In high-dimensional spaces, Euclidean distance suffers from the 'curse of dimensionality,' where contrast between near and far points diminishes. Cosine distance remains robust because it measures angular separation on the unit hypersphere rather than absolute spatial distance. For text data, two documents with identical word distributions but different lengths will have a large Euclidean distance but a cosine distance of zero. This makes cosine the default choice for LIME's exponential kernel weighting, ensuring that perturbed samples are weighted by their semantic orientation relative to the original instance.

LIME uses an exponential kernel on cosine distance to assign weights to perturbed samples: weight = exp(-distance² / kernel_width²). When cosine distance is small (high similarity), the weight approaches 1.0, giving maximum influence to samples that are semantically close to the original instance. As the distance increases toward 1 (orthogonal vectors), the weight decays rapidly, enforcing strict locality. The kernel width hyperparameter controls this decay rate, directly balancing the bias-variance trade-off of the local surrogate model.

Cosine distance is particularly well-suited for non-negative feature spaces like TF-IDF or bag-of-words representations, where all vector components are ≥ 0. In this constrained positive orthant, the cosine similarity is bounded between 0 and 1, eliminating the possibility of negative correlations. This simplifies interpretation: a similarity of 0 means the documents share no terms, while 1 means identical term distributions. This property ensures that LIME's neighborhood sampling in text domains always produces meaningful, positively correlated distance measurements.

The foundational text representation that cosine distance operates on. Term Frequency-Inverse Document Frequency transforms raw text into sparse numerical vectors by weighting each term's importance: frequent terms in a document get higher scores, but terms common across the entire corpus are penalized. This ensures that discriminative words like 'neural' carry more weight than stop words like 'the.' In LIME, TF-IDF vectors serve as the interpretable representation for text data before perturbation sampling and surrogate model training.

The complement of cosine distance, measuring the cosine of the angle between two vectors rather than the angular separation itself. Cosine similarity ranges from -1 (completely opposite) to 1 (identical direction), with 0 indicating orthogonality. The relationship is direct: cosine_distance = 1 - cosine_similarity. While similarity is intuitive for measuring relatedness, distance is preferred in LIME's exponential kernel for weighting perturbed samples, as it provides a proper metric where smaller values indicate closer proximity.

A critical distinction in high-dimensional text analysis. Euclidean distance measures the straight-line magnitude between vector endpoints, making it sensitive to document length—a 500-word article and a 5000-word treatise on the same topic would appear far apart. Cosine distance ignores magnitude entirely, focusing solely on orientation. This length-invariance property makes it the default choice for text comparison in LIME, where two documents with identical word distributions but different lengths should be treated as nearly identical.

The mechanism that converts cosine distances into sample weights for LIME's local surrogate model. The exponential kernel applies the formula weight = exp(-distance² / kernel_width²), where kernel width controls the locality radius. Samples with small cosine distances (high similarity) receive weights near 1.0, while distant perturbations approach zero weight. This enforces the locality constraint: the interpretable model fits the black-box behavior most accurately in the immediate neighborhood of the instance being explained.

The computational reality of text vectors that makes cosine distance efficient. Unlike dense embeddings from neural networks, TF-IDF vectors are sparse—most dimensions are zero because any given document contains only a tiny fraction of the corpus vocabulary. Cosine distance exploits this sparsity: the dot product computation A·B only needs to iterate over non-zero entries shared between both vectors. For LIME's perturbation sampling, where thousands of synthetic documents are generated and compared, this sparsity ensures the explanation pipeline remains computationally tractable.

The geometric interpretation of cosine distance as the arc length on a unit hypersphere. When vectors are L2-normalized to unit length, cosine distance becomes proportional to the angle between them in radians. This normalization step is implicit in cosine distance calculations and explains why document length differences vanish: all vectors are projected onto the surface of a unit sphere, where only direction matters. In LIME, this ensures that perturbed samples with different token counts but similar word proportions cluster together in the local neighborhood.