A Systematic Comparison of Single-Cell Perturbation Response Prediction Models

Predicting single-cell transcriptomes following perturbation is crucial for understanding gene regulation and guiding drug discovery. Yet, the complexity of perturbation effects pose significant challenges for robust predictive modeling. Although recent efforts have introduced foundation models alongside traditional statistical and machine learning approaches, a comprehensive benchmarking study of their predictive performance has been lacking. Here we establish a standardized evaluation framework and systematically assess 9 single-cell perturbation prediction models on 17 diverse datasets, spanning multiple cell types and perturbation modalities. Leveraging a multifaceted suite of 24 evaluation metrics, we find that models often excel in capturing global expression profiles yet struggle to predict the nuanced effects of perturbed genes, or vice versa. Moreover, while foundation models frequently outperform simpler methods, they tend to converge on population averages, struggling to capture heterogeneous cellular responses to perturbations. Taken together, our study highlights both the potential and limitations of single-cell foundation models, identifies opportunities for future improvement, and provide a roadmap for advancing single-cell predictive biology.