Graph Neural Network Fusion

The foundational step where disparate data modalities are transformed into a unified graph structure. Image patches become nodes connected by spatial adjacency edges, genomic variants link to genes via functional relationships, and clinical concepts connect through ontological hierarchies from SNOMED CT. Cross-modal edges are established using prior knowledge—such as linking a radiological finding to a known genetic marker—or learned dynamically through attention mechanisms. This construction defines the inductive bias that guides all downstream learning.

The core computational mechanism where nodes iteratively aggregate and transform information from their neighbors. Each GNN layer applies a message function to compute information from adjacent nodes and an update function to revise the target node's hidden state. After multiple layers, each node's embedding captures information from its multi-hop neighborhood. For a tumor node, this means its final representation encodes not just its own imaging features but also the functional impact of connected genomic mutations and relevant patient history.

Unlike CNNs that assume a regular grid structure or transformers that learn all pairwise interactions, GNNs explicitly leverage known relationships as a strong structural prior. This makes them highly data-efficient in medical domains where annotated multi-modal datasets are scarce. By hard-coding that a specific imaging phenotype is related to a particular gene expression pathway, the model does not need to learn this association from scratch, reducing overfitting and improving generalization to rare disease presentations.

After message passing, a readout function aggregates all node embeddings into a single, fixed-size graph-level representation. Common approaches include:

• Global pooling: averaging or summing all node states
• Set2Set: a learned, attention-based aggregation
• Virtual node: a dedicated super-node connected to all others

This graph embedding serves as the Holistic Patient Representation, directly feeding into downstream tasks like survival prediction or treatment response classification.

Beyond node classification, GNN fusion enables link prediction to discover previously unknown relationships between modalities. By training on known imaging-genomic correlations, the model learns to score the likelihood of an edge between an uncharacterized radiological feature and a genetic variant. This capability transforms the diagnostic model from a passive classifier into an active hypothesis generation engine, suggesting novel radiogenomic biomarkers for further clinical validation.

A specialized architecture for graphs with multiple node and edge types. Type-specific attention learns the importance of different neighbors—a genomic node may attend more strongly to other genomic nodes than to imaging nodes in one context, and vice versa in another. Semantic-level attention then aggregates information across different meta-paths (e.g., Image→Gene→Disease vs. Image→Clinical Finding→Disease). This hierarchical attention provides inherent Multimodal Explainability, producing interpretable importance weights for each modality and relationship.

The process of defining nodes (patients, genes, image patches) and edges (correlations, spatial proximity, co-occurrence) to structure a diagnostic problem as a graph.

• Node Features: Embeddings from a CNN for an image patch or a genomic assay vector.
• Edge Types: Can represent physical proximity in a tissue sample, regulatory relationships in a gene network, or temporal sequences in clinical events.
• Heterogeneous Graphs: Support multiple node and edge types simultaneously, allowing a single graph to model imaging, text, and omics data.

The core computational mechanism of a GNN where nodes iteratively aggregate feature information from their neighbors to update their own state.

• Aggregate: A node collects feature vectors from its immediate neighbors.
• Combine: A neural network function (e.g., a linear layer + activation) merges the aggregated neighbor information with the node's own current state.
• Readout: After multiple message-passing layers, a global pooling operation aggregates all node states into a single graph-level vector for final classification.

An advanced GNN layer that introduces a self-attention mechanism to weigh the importance of different neighboring nodes during message passing.

• Dynamic Weighting: Instead of treating all neighbors equally, the model learns an attention coefficient for each edge.
• Cross-Modal Relevance: In a diagnostic graph, GAT can learn to automatically down-weight noisy imaging features and up-weight critical genomic markers when fusing data for a specific patient.
• Interpretability: The learned attention weights can be visualized to show which data points (e.g., a specific lesion or gene) most influenced the prediction.

A GNN architecture theoretically proven to be as powerful as the Weisfeiler-Lehman (WL) graph isomorphism test, making it maximally expressive for distinguishing different graph structures.

• Injective Aggregation: Uses a multi-layer perceptron (MLP) to combine node and neighbor features, ensuring unique mappings for different neighborhood structures.
• Structural Sensitivity: Ideal for capturing complex, non-linear relational topologies in biological networks, such as protein-protein interaction graphs or intricate disease pathways.
• Diagnostic Application: Excels at tasks where the global connectivity pattern, not just local features, defines the diagnosis.

A specialized GNN designed to operate on heterogeneous graphs with multiple types of directed edges (relations).

• Relation-Specific Weights: Maintains a distinct weight matrix for each relation type (e.g., 'has_symptom', 'expresses_gene', 'located_in').
• Knowledge Graph Fusion: Naturally suited for integrating a medical knowledge graph (like SNOMED CT) with patient-specific data nodes.
• Basis Decomposition: Uses a linear combination of basis matrices to prevent parameter explosion when the number of relation types is large, ensuring the model remains scalable.

An architecture that models both spatial dependencies (via graph structure) and temporal dynamics (via sequence models) simultaneously.

• Spatial Graph: A fixed graph representing physical sensor locations or anatomical regions.
• Temporal Component: A 1D CNN or RNN processes the sequence of graph snapshots over time.
• Clinical Use Case: Analyzing the progression of a tumor by modeling each MRI slice as a node in a spatial graph and each follow-up scan as a time step, fusing structural changes with temporal growth patterns.