Measurement Reliability of Individual Differences in Naturalistic Reading

<p>Research on individual differences in reading often links participant-specific effects of linguistic predictors to reader characteristics such as cognitive abilities. Such analyses require that these effects be measured reliably, yet reported reliabilities (operationalized as correlations across measurement contexts) vary widely, raising concerns about the suitability of these effects for individual-differences research. We argue that reliability should first be understood as a measurement outcome shaped by experimental design and estimation method before drawing conclusions about the inherent reliability of a specific linguistic predictor, task, or paradigm. Using formal derivations and simulations, we show that, under conventional correlation estimation methods, reliability is attenuated by measurement noise, limited trial counts, low predictor variability, and weak between-subject variance, while participant count primarily determines estimation uncertainty. By comparing five estimation methods, we identify Bayesian two-task hierarchical models that estimate the latent correlation as providing the most accurate estimates and the best-calibrated uncertainty. We validate these findings using cross-session eye-tracking data from the German Individual Differences Corpus (InDiCo), focusing on individual effects of word length, lexical frequency, and surprisal on gaze duration. Empirical reliability estimates increase with trial count, participant count, predictor variability, and the ratio of between-subject variance to measurement noise, with the Bayesian latent-correlation approach performing the best overall. These findings provide a principled framework for interpreting observed reliability, evaluating estimation methods, and improving the planning and designing of studies on individual differences in naturalistic reading.</p>