MonoBERT

MonoBERT treats re-ranking as a binary classification task, evaluating each query-document pair independently. The model takes a concatenated [CLS] query [SEP] document [SEP] sequence and uses the final hidden state of the [CLS] token as the aggregate representation. A single linear layer projects this representation to a logit, which is passed through a sigmoid function to produce a relevance probability between 0 and 1. This pointwise approach ignores the relative ordering of other candidates, making it computationally predictable but blind to list-level context.

Unlike Bi-Encoders that encode queries and documents independently, MonoBERT enables token-level cross-attention across the entire concatenated sequence. Every query token attends to every document token simultaneously through the transformer's self-attention layers. This allows the model to capture:

• Exact lexical matches between query and passage terms
• Semantic paraphrasing where different words share contextual meaning
• Term importance weighting based on mutual relevance This rich interaction is the primary source of MonoBERT's precision gains over dual-encoder architectures.

MonoBERT is typically initialized from a pre-trained BERT-base checkpoint and fine-tuned on the MS MARCO passage ranking dataset. The training objective is a standard cross-entropy loss against binary relevance labels, where passages marked as relevant by human annotators are positive examples and all others are negatives. Key training dynamics include:

• Hard negative mining to expose the model to difficult distractors
• In-batch negatives to increase training efficiency without explicit sampling
• A maximum sequence length of 512 tokens, with longer documents truncated This fine-tuning adapts BERT's general language understanding to the specific task of passage relevance judgment.

MonoBERT's precision comes at a significant inference latency cost. Each query-document pair requires a full forward pass through all 12 transformer layers, making it impractical for first-stage retrieval over millions of documents. Typical deployment patterns:

• Applied only to top-k candidates (k=100 to 1000) from a fast Bi-Encoder retriever
• Latency ranges from 10-50ms per pair on GPU, depending on batch size and sequence length
• Cannot be pre-computed because the query participates in every attention operation This cost profile necessitates a cascade ranking architecture where MonoBERT serves as the precision-focused final stage.

The raw logits from MonoBERT's classification head are not inherently well-calibrated as absolute relevance probabilities. The model tends toward overconfident predictions, with scores clustering near 0 or 1. Production deployments often apply:

• Temperature scaling to soften the sigmoid output
• Platt scaling to fit a logistic calibration curve on a held-out validation set
• Min-max normalization across the candidate set for a given query Proper calibration is critical when MonoBERT scores feed into downstream rank aggregation methods like Reciprocal Rank Fusion (RRF) or when combining with sparse lexical scores like BM25.

MonoBERT frequently serves as the teacher model in Cross-Encoder distillation pipelines. Its full-attention scoring distribution captures nuanced relevance signals that can be transferred to a lightweight Bi-Encoder student. The distillation process typically uses:

• KL divergence loss between the teacher's softmax distribution and the student's output
• Margin-based distillation where the student learns to preserve the score gap between positive and hard negative pairs
• Training on large-scale query logs where MonoBERT provides pseudo-relevance labels This enables the student Bi-Encoder to approximate MonoBERT's precision while maintaining the efficiency required for first-stage retrieval over millions of documents.