CASP (Critical Assessment of Structure Prediction)

CASP operates on a strict double-blind principle. Organizers solicit recently solved but unpublished structures from experimentalists. Computational groups then submit predictions before the experimental coordinates are released. This eliminates any possibility of overfitting or data leakage, providing an unvarnished assessment of a method's true generalization capability on novel targets.

CASP evaluates predictions across several distinct modeling categories to benchmark different aspects of the folding problem:

• Tertiary Structure Prediction (TS): Modeling the 3D coordinates of single chains.
• Quaternary Structure Prediction (QS): Modeling the assembly of multi-chain complexes.
• Estimation of Model Accuracy (EMA): Predicting the local and global errors in one's own models.
• Data-Assisted Modeling: Leveraging sparse experimental data like SAXS or crosslinking.

Predictions are scored against the newly released experimental structures using standardized, objective metrics. The primary measure is the Global Distance Test-Total Score (GDT-TS), which evaluates the similarity of backbone topology. Other critical metrics include AL0 (Accuracy of Local 0-4Å) for side-chain precision and lDDT (local Distance Difference Test) for per-residue accuracy, ensuring a holistic evaluation beyond simple RMSD.

Each prediction category is analyzed by an independent, expert assessor who has no competing entries. The assessor performs deep statistical analysis on the results, identifies the specific strengths and failure modes of different algorithmic approaches, and publishes a comprehensive, citable paper that defines the state of the field. This expert narrative contextualizes the raw scores.

The CASP format is not static; it evolves based on community feedback and emerging challenges. Recognizing that a single static model is insufficient, CASP introduced the Ensembles category to evaluate methods that predict conformational flexibility. Similarly, the Data-Assisted category was created to bridge the gap between purely computational and integrative structural biology, constantly pushing the frontier of what is assessed.

CASP serves as the definitive historical record of progress in the field. The most famous result is AlphaFold2's performance at CASP14 (2020), where it achieved median GDT-TS scores competitive with experimental methods, effectively solving the single-chain protein folding problem. This landmark event, validated by CASP's blind rigor, catalyzed the current revolution in AI-driven structural biology.

The primary evaluation metric used in CASP to assess prediction accuracy. GDT_TS measures the percentage of Cα atoms in the predicted model that fall within specified distance thresholds of the experimental structure after optimal superposition.

• Calculated across four distance thresholds: 1, 2, 4, and 8 Å
• Ranges from 0 to 100, with higher scores indicating better predictions
• Less sensitive to local outliers than RMSD
• Considered the gold standard for CASP ranking

CASP evaluates predictions across several distinct categories to benchmark different aspects of structure prediction capability:

• TBM (Template-Based Modeling): Targets with detectable homologs in the PDB
• FM (Free Modeling): Targets with no known structural templates, testing ab initio prediction
• TBM/FM: Borderline difficulty targets
• QA (Quality Assessment): Evaluating the accuracy of predicted models without knowing the experimental structure
• Data-assisted: Predictions incorporating sparse experimental data like chemical crosslinks or SAXS

CASP operates as a double-blind experiment to ensure unbiased evaluation. Organizers solicit newly solved but unpublished experimental structures from structural biologists worldwide.

• Predictors submit models before the experimental structures are released
• Assessors independently evaluate predictions against the withheld experimental data
• Results are revealed and discussed at the biennial CASP meeting in December
• This format has driven objective progress measurement since CASP1 in 1994

CASP15 (2022) marked a paradigm shift where the field moved beyond single-chain prediction to new frontiers:

• Protein complexes: Assessing quaternary structure and oligomeric assembly prediction
• RNA structure prediction: New category for RNA 3D structure modeling
• Conformational ensembles: Evaluating predictions of multiple functional states
• Ligand docking: Assessing protein-ligand complex prediction accuracy
• The focus shifted from 'can we predict structure?' to 'can we predict dynamics, interactions, and function?'