TyDi QA

TyDi QA includes 11 languages selected to maximize morphological and syntactic diversity, not just geographic coverage. Languages range from agglutinative (Finnish, Turkish) to isolating (Indonesian) to polysynthetic (Swahili). This forces models to learn genuine cross-lingual understanding rather than relying on shared Indo-European structures. The selection includes Arabic, Bengali, English, Finnish, Indonesian, Japanese, Kiswahili, Korean, Russian, Telugu, and Thai.

The benchmark's core innovation is the confounding of lexical cues. In standard QA datasets, a question word like 'When' often correlates with a date in the answer. TyDi QA explicitly designs questions where the answer type cannot be predicted from the question's lexical form alone. For example, a 'What' question might require a person's name as an answer, preventing models from using simple part-of-speech heuristics to guess answer spans.

Every question in TyDi QA is verified to be answerable from a single, provided Wikipedia passage. Human annotators first wrote questions based on a passage, then verified that the answer was explicitly stated. This eliminates the confounding variable of unanswerable questions found in other datasets, allowing researchers to isolate and measure a model's pure reading comprehension and cross-lingual transfer ability without ambiguity about whether the information exists.

TyDi QA uses a two-stage annotation process. First, annotators create candidate questions in their native language prompted by a passage. Second, a separate set of annotators provides minimal-span answers. This decoupling ensures that questions are natural and not biased by the annotator's knowledge of the answer span. The pipeline also collects passage-level language metadata, ensuring all context is in the target language without code-switching.

The primary evaluation protocol is zero-shot transfer: models are trained exclusively on English data (SQuAD or Natural Questions) and tested directly on the 10 non-English TyDi QA languages. This measures a model's language-agnostic representation quality. A high score indicates that the model's internal understanding of question-answer semantics is decoupled from the surface form of English, a critical capability for deploying QA systems in low-resource languages.

TyDi QA provides two evaluation tracks. The Gold Passage (GoldP) task provides the model with the single correct Wikipedia passage containing the answer. The Full-Wiki (Wiki) task requires the model to first retrieve the correct passage from the entire Wikipedia corpus for that language before extracting the answer. This dual setup allows researchers to independently benchmark a model's reading comprehension and its cross-lingual retrieval capabilities.

TyDi QA's primary design goal is to include languages that are structurally distinct from English, avoiding the common bias toward Indo-European languages. It covers 11 languages selected for their diversity in word order, morphology, and writing systems.

• Included Languages: Arabic, Bengali, English, Finnish, Indonesian, Japanese, Kiswahili, Korean, Russian, Telugu, Thai
• Key Feature: Prevents models from succeeding through simple lexical overlap, forcing genuine cross-lingual understanding
• Typological Spread: Includes agglutinative (Finnish), isolating (Thai), and polysynthetic characteristics

TyDi QA defines two distinct answer selection tasks to evaluate model precision. The Primary Task requires selecting the single minimal span that answers the question, while the Minimal Answer Span task focuses on extracting the most concise possible answer.

• Gold Passage Corpus: Human-annotated passages provide the ground truth context
• Answerability Judgment: Models must also predict if a question is answerable from the given passage
• Evaluation Metric: Uses F1 score at the passage level, penalizing models that cannot locate the precise answer boundary

All questions in TyDi QA are grounded in Wikipedia articles, ensuring factual verifiability. Questions were written by native speakers who were prompted to create queries that a monolingual English speaker could not answer without translation.

• Human-Generated: Questions are not machine-translated, preserving natural phrasing and idiomatic usage
• Lexical Overlap Minimization: Annotators were explicitly instructed to avoid copying words directly from the passage
• Quality Control: Multiple rounds of validation ensure answers are unique and unambiguous

The Gold Passage (GoldP) variant isolates the reading comprehension challenge by providing a single, guaranteed-relevant Wikipedia passage. This decouples retrieval quality from comprehension ability.

• Task: Given a question and a gold passage, extract the minimal answer span
• Baseline Models: mBERT and XLM-RoBERTa fine-tuned on multilingual SQuAD-style data
• Key Insight: Even with perfect retrieval, significant performance gaps exist between high-resource and low-resource languages

TyDi QA serves as a critical benchmark for zero-shot cross-lingual transfer. Models are typically fine-tuned on English QA data (like SQuAD) and evaluated directly on TyDi QA's target languages without any target-language training data.

• Model Architectures: XLM-RoBERTa, mT5, and ByT5 are standard baselines
• Performance Gap: Zero-shot transfer F1 scores can be 20-30 points lower than monolingual English performance
• Data Augmentation: Translate-train approaches, where English data is machine-translated, significantly boost low-resource performance

TyDi QA complements other multilingual evaluation suites by focusing specifically on typological diversity and non-English-native question formulation. It is often used alongside MLQA and XQuAD.

• MLQA: Tests cross-lingual QA where questions are in English but context is in a target language
• XQuAD: A professional translation of SQuAD into 11 languages, testing translation robustness
• MKQA: An open-domain QA benchmark with queries in 26 languages, testing end-to-end retrieval and comprehension