Abstract
<p>Identifying good forecasters, individuals who consistently make accurate predictions about future events, is one central research topic in the forecasting literature. One of the latest developments in this stream of research is the Forecasting Proficiency Test (FPT), which aims to provide a reliable and accessible measure of individual forecasting ability. A large portion of the FPT consists of quantile elicitation format (QEF) forecasts rather than the more common probability elicitation format (PEF). We propose a novel psychometric model that draws upon item response theory, with the goal of measuring item features of QEF items and capturing individual forecasting ability as a latent construct. In doing so, we outline important theoretical differences between latent constructs and accuracy metrics commonly employed in the forecasting literature to measure forecasting proficiency. We demonstrate the practical usefulness of the proposed model by applying it to the QEF questions of the FPT, where we find that it captures meaningful item features and adequately measures forecasting ability. Namely, the estimated latent construct forecasting ability showed a Pearson correlation of r = .85 with measures based on strictly proper scoring rules of forecasting accuracy. We conclude by discussing the practical benefits of measuring item parameters of forecasting items, the specific limitations of psychometric models applied to forecasting items, and when and where we believe they are most appropriate in the forecasting domain.</p>