Abstract
<jats:p>Background: Retrospective intensive care unit prediction models are vulnerable to temporal leakage when predictors include information recorded after the intended prediction time. We developed and externally validated an ICU hour-24 landmark prediction pipeline for in-hospital mortality among critically ill patients with coronary artery disease using timestamp-restricted features from MIMIC-IV and eICU. Methods: Adults with coronary artery disease or coronary heart disease who were alive and remained under ICU observation at hour 24 were included. MIMIC-IV was used for model development, and eICU was reserved for external validation. Dynamic events were restricted to ICU admission through hour 24 in MIMIC-IV and offsets of 0-1440 minutes in eICU before aggregation. We evaluated an XGBoost model using baseline and respiratory-support predictors and a 102-predictor random forest using baseline, respiratory-support, and treatment predictors. Robustness was assessed across 30 repeated patient-grouped MIMIC-IV validation splits, and external uncertainty was estimated using 1,000 subject-clustered bootstrap resamples. Results: The MIMIC-IV cohort included 4,341 ICU stays with 993 deaths, and the eICU cohort included 19,464 stays with 2,237 deaths. In eICU, XGBoost achieved a ROC-AUC of 0.7973 (95% CI, 0.7878-0.8068), a PR-AUC (calculated as average precision) of 0.3627 (95% CI, 0.3423-0.3843), and a Brier score of 0.1189 (95% CI, 0.1164-0.1213). The random forest achieved a ROC-AUC of 0.8060 (95% CI, 0.7960-0.8154), a PR-AUC of 0.3687 (95% CI, 0.3471-0.3915), and a Brier score of 0.1032 (95% CI, 0.1012-0.1054). The random forest had a modestly higher ROC-AUC and lower Brier score than XGBoost. Mean validation ROC-AUCs across repeated MIMIC-IV splits were 0.7970 and 0.7954, respectively. Exploratory analyses suggested that narrower, consistently harmonized feature sets transported more reliably than broader expansions. Conclusions: Timestamp-restricted first-day models achieved external ROC-AUCs of approximately 0.80. However, external calibration remained imperfect, and local recalibration and prospective evaluation would be required before clinical use.</jats:p>