Perceiver IO

The core innovation. A fixed-size set of learned latent vectors queries the input byte array via cross-attention. This compresses arbitrarily large inputs (e.g., a 50-megapixel whole slide image) into a compact representation. The bulk of the Transformer computation then operates only on this small latent set, not the raw input. This decoupling means the computational cost of the self-attention trunk is independent of input size, a critical advantage over standard Vision Transformers for gigapixel pathology.

Unlike the original Perceiver which produced a single global output, Perceiver IO uses a second cross-attention module to decode from the latent array into an output space of arbitrary size and structure. By constructing a specific output query array, the same architecture can produce:

• A single class label for image classification
• A dense pixel-level map for medical image segmentation
• A sequence of tokens for multimodal captioning
• Bounding box coordinates for object detection This makes it a genuinely unified architecture for diverse diagnostic tasks.

Perceiver IO uses Fourier feature position encodings rather than learned or fixed sinusoidal encodings. Input element positions (e.g., pixel coordinates, time indices) are projected through sine and cosine functions of logarithmically spaced frequencies. This provides a smooth, continuous representation of position that generalizes to coordinates unseen during training. For medical imaging, this allows the model to handle variable-resolution DICOM scans and 3D volumetric data without retraining, as the positional signal is a continuous function of spatial location.

All inputs are treated as a flat set of byte arrays with associated modality-specific position encodings. Images become a set of pixel-value-position tuples. Text becomes a set of token-index pairs. Point clouds become 3D coordinates with features. This radical unification means a single Perceiver IO model can be trained on multi-modal diagnostic data—fusing CT volumes, clinical notes, and genomic sequences—without modality-specific encoders. The architecture learns to disentangle modalities through the latent bottleneck.

Because the latent array has a fixed size (typically 256-1024 vectors), the self-attention cost is O(L²) where L is the latent size—a constant. The cross-attention layers scale O(M × L) where M is the input size. This is linear in input size, not quadratic. For a 100,000 × 100,000 pixel pathology whole slide image, a ViT would be computationally infeasible. Perceiver IO processes it with linear cost, making it viable for gigapixel computational pathology on commodity hardware.

The latent array is not a single-pass compression. It is processed through multiple self-attention layers in the latent Transformer trunk, iteratively refining its representation of the input. This allows the model to perform complex relational reasoning within the compressed space. Each layer can re-weight which input elements to attend to via the cross-attention weights, effectively implementing an iterative focusing mechanism that mimics how a radiologist might repeatedly scan different regions of an image before forming a diagnosis.