Abstract
<title>Abstract</title> <p> Prediction models are widely used to forecast outcomes and support decision-making across diverse applications. However, many modern prediction models are inherently non-deterministic, meaning that repeated training on identical data can yield different predictions and potentially inconsistent decisions. Assessing <italic>prediction stability</italic> is therefore important for evaluating the reliability of such models. Existing approaches to quantifying prediction stability were primarily developed for repeated fits of random forest models using the intraclass correlation coefficient (ICC) for metric predictions and Fleiss’ \((\kappa)\) for categorical predictions. However, these measures have important limitations: they do not naturally extend to ordinal predictions, require multiple test objects, and depend on the structure of the test data, potentially confounding prediction stability with data characteristics. We therefore propose a general framework for quantifying prediction stability in non-deterministic prediction models. Specifically, we introduce the <italic>data-based</italic> and <italic>model-based prediction stability</italic> , defined as one minus the ratio of observed to expected instability estimated from training data or model predictions. The proposed measures are applicable to metric, categorical, and ordinal predictions, can be evaluated for individual test objects, and are substantially less sensitive to the composition of the test data. The framework is implemented in the R package \texttt{APS} for practical use. </p>