Abstract
<p>Interventions aimed at improving appropriate reliance on AI have had mixed effectiveness. We argue that this may be due to the lack of attention to individual differences in human decision strategies, combined with overconfidence in relying on the results of aggregate-level analyses. We approach this issue in a set of three studies. In Study 1, we design a set of decision strategies grounded in self- and AI-confidence to provide a robust first-order approximation of what individual strategies can be. We use simulation to assess the conditions under which these strategies are identifiable. In Study 2, using a face-matching task, we measured whether participants adjusted their initial decisions when an AI model advised the opposite response with varying levels of confidence. Group-level analyses suggest that both initial self-confidence and AI confidence are strong predictors of human reliance on AI advice. However, when assessing the strategies at the individual level, we find evidence for the fact that a sizable subset of participants did not attend to both confidence scores, contradicting the group-level analyses. Using simulation (Study 3), we show that group-level analyses can easily paint a picture of the underlying strategies that is at odds with the actual underlying distribution of individual strategies. Taken together, our results underscore the critical role of recognizing individual human decision strategies in dealing with AI to better understand and support human-AI decision-making.</p>