Nested cross-validation reduces optimism in metabolomics-based prediction of immune checkpoint inhibitor outcomes

<title>Abstract</title> <p>Immune checkpoint inhibitors (ICIs) benefit only a subset of cancer patients, creating a need for predictive biomarkers. Blood metabolomics is an attractive minimally invasive approach; however, most datasets are small and high-dimensional, making apparent model performance sensitive to internal validation design. We evaluated how single-split holdout, non-nested cross-validation (CV), and nested cross-validation (nCV) affect discrimination and uncertainty in metabolomics-based prediction of ICI outcomes. The analysis included six pre-ICI and ten post-ICI liquid chromatography/mass spectrometry (LC/MS) datasets, plus one pre-ICI nuclear magnetic resonance (NMR) dataset. In pre-ICI LC/MS datasets, mean area under the receiver operating characteristic curve (ROC-AUC) was 0.705 for nCV, 0.882 for non-nested CV, and 0.601 for holdout; corresponding values in post-ICI LC/MS datasets were 0.646, 0.838, and 0.639. Non-nested CV overestimated ROC-AUC relative to nCV by mean differences of 0.178 and 0.192 in pre- and post-ICI datasets, respectively. Most optimism arose from feature ranking, and nCV yielded narrower bootstrap 95% confidence intervals than holdout. The independent NMR dataset showed the same qualitative pattern. These findings support nCV as the default internal validation strategy for metabolomics prediction pipelines requiring model selection, while external validation remains essential before clinical use.</p>