Entity Co-occurrence

Co-occurrence is measured using metrics like Pointwise Mutual Information (PMI) and Normalized Google Distance (NGD). PMI quantifies the deviation from statistical independence: a high positive PMI indicates entities appear together more often than chance. NGD leverages search engine result counts to approximate semantic distance. These scores are the raw input for building co-occurrence matrices that underpin distributional semantics.

The context window—the span of tokens within which co-occurrence is measured—critically shapes the resulting relationships. A narrow window (e.g., 5 words) captures syntactic and functional relationships (e.g., 'CEO' and 'announced'). A wider window or full-document scope captures topical and thematic relatedness (e.g., 'Tesla' and 'electric vehicle'). The window size is a tunable hyperparameter that trades syntactic precision for semantic breadth.

Co-occurrence statistics are a primary signal for unsupervised knowledge base population. When two entities co-occur with high frequency and statistical significance, an edge is hypothesized between their corresponding nodes in a knowledge graph. This edge can later be labeled with a specific relation type via relation extraction models. This process transforms unstructured text into structured semantic triples.

Co-occurrence is distinct from coreference resolution. Coreference identifies when two textual mentions (e.g., 'Apple Inc.' and 'it') refer to the same real-world entity. Co-occurrence identifies when two distinct entities appear in the same context. Confusing these leads to incorrect knowledge graph edges. A system must resolve coreference before calculating co-occurrence to avoid linking an entity to its own pronoun.

Co-occurrence signals power multiple downstream tasks:

• Entity Linking Disambiguation: A mention's surrounding co-occurring entities provide strong contextual cues for selecting the correct knowledge base entry.
• Topic Modeling: Co-occurrence patterns across documents are the basis for algorithms like Latent Dirichlet Allocation (LDA).
• Search Ranking: Modern search engines use co-occurrence to understand query intent and document relevance beyond exact keyword matching.

Raw co-occurrence is noisy. High frequency can stem from common popularity rather than genuine semantic relatedness (e.g., 'Google' and 'Monday' co-occur often but are not meaningfully related). Negative sampling and significance testing are required to filter spurious correlations. Furthermore, co-occurrence alone cannot specify the nature of a relationship—it signals that a relationship exists, not what it is, necessitating downstream relation classification.

A statistical measure that quantifies the association strength between two entities by comparing their observed co-occurrence probability against their expected probability if they were independent. Unlike raw frequency counts, PMI normalizes for marginal entity frequency, preventing common entities from dominating association scores.

• Positive PMI: Entities co-occur more often than chance (genuine semantic association)
• Zero PMI: Entities are statistically independent
• Negative PMI: Entities co-occur less than expected (potential mutual exclusion)

Used extensively in word embedding models like word2vec and GloVe to construct co-occurrence matrices before dimensionality reduction.

A square matrix where rows and columns represent unique entities and each cell contains the frequency count of how often two entities appear together within a defined context window. This matrix serves as the raw input for many dimensionality reduction techniques.

• Context Window: Typically 2-10 words on either side of a target entity
• Sparsity: Most entity pairs never co-occur, requiring sparse matrix representations
• Weighting: Raw counts are often transformed using PMI or TF-IDF before downstream use

Co-occurrence matrices are foundational to distributional semantics, where an entity's meaning is defined by the company it keeps.

The linguistic principle stating that entities appearing in similar contexts tend to have similar meanings. First articulated by Zellig Harris (1954) and popularized by John Rupert Firth's dictum: "You shall know a word by the company it keeps."

This hypothesis provides the theoretical justification for entity co-occurrence analysis. If 'Tesla' and 'Rivian' frequently co-occur with 'electric vehicle,' 'battery range,' and 'autonomous driving,' they occupy similar positions in semantic space.

Modern transformer architectures operationalize this principle through self-attention mechanisms that dynamically weight co-occurrence relationships.

Algorithms that treat entities as nodes and co-occurrence relationships as weighted edges in a semantic network. TextRank and LexRank adapt PageRank's recursive importance scoring to identify the most central entities in a document.

• Node Weight: Initialized by frequency, iteratively updated by neighbor importance
• Edge Weight: Co-occurrence count or PMI score between connected entities
• Convergence: Algorithm runs until node scores stabilize below a threshold

These methods extract salient entities without requiring supervised training data, making them ideal for unsupervised keyphrase extraction and document summarization.

The deliberate design of the span of tokens within which co-occurrence is measured. Window size and shape dramatically affect which entity relationships are captured.

• Narrow windows (2-3 words): Capture syntactic relationships and fixed phrases like 'San Francisco'
• Wide windows (10+ words): Capture topical and thematic associations like 'machine learning' with 'GPU'
• Directional windows: Asymmetric windows that only count entities appearing after the target, useful for predictive tasks
• Sentence boundaries: Respecting sentence limits prevents spurious cross-sentence associations

Optimal window engineering depends on whether the goal is syntactic parsing or semantic topic modeling.

The principle that an entity's importance within a document correlates with the number and strength of its co-occurrence links to other entities. An entity mentioned in isolation carries less weight than one densely connected in the document's entity graph.

• Degree Centrality: Number of unique co-occurring entities
• Weighted Degree: Sum of co-occurrence frequencies or PMI scores
• Betweenness Centrality: How often an entity serves as a bridge between other entity clusters

This approach underpins salience scoring for AI-generated summaries, where models must identify which entities to retain when condensing text. Google's entity-oriented search uses similar principles to determine which entities are central to a page's topic.