Abstract
<title>Abstract</title> <p>Rather than asking which model performs best, this study asks a more fundamental question: under what conditions should complex models be expected to outperform simpler alternatives in data-limited scientific machine learning? We address this question across two empirical case studies — kinase inhibitor binding affinity prediction and ESOL aqueous solubility prediction — and introduce a preliminary Three-Principle Generalization Framework grounded in the Generalization Failure Triad. We evaluated four model classes — linear regression, ridge regression, XGBoost, and partial least squares (PLS) — on a dataset of 100 kinase inhibitor molecules with experimentally determined binding affinities across 14 protein targets. Linear regression achieved the highest performance (R² = 0.9865, RMSE = 0.1307), while external validation on 60 independent molecules demonstrated robust generalization (R² = 0.9385, ΔR² = 0.0431). The external validation set was locked before any model training, feature selection, or hyperparameter tuning. From these results, we propose a framework grounded in three theoretical principles: (1) Model Capacity — whether the model class is appropriately sized for the available sample (n/p < 10); (2) Estimation Stability — whether performance is consistent across training partitions (CV SD > 0.05); and (3) Out-of-Sample Transferability — whether learned patterns hold on unseen data (ΔR² > 0.08). Each threshold is presented as a first empirical instantiation requiring validation on independent datasets. All three criteria consistently identify linear models as the appropriate choice in the present study. The objective of this study is to provide practitioners with an empirically motivated decision framework for model selection in data-limited scientific settings. These findings provide evidence that model simplicity should remain an important baseline consideration in small-sample scientific machine learning.</p>