A Multimodal Transformer Framework with State-Aware Adaptive Fusion for Ultra-Short-Term Solar Irradiance Forecasting with Sky Cloud Images

<title>Abstract</title> <p>As renewable energy penetration continues to increase, accurate ultra-short-term solar irradiance forecasting is essential for improving energy utilization and supporting power system operation. However, irradiance variations are jointly influenced by historical trends and rapid cloud changes, making traditional single-modal methods insufficient for capturing complex dynamics under varying weather conditions. This paper proposes a multimodal solar irradiance forecasting framework based on the collaborative representation of sky cloud images and numerical sequences. The framework introduces a heterogeneous modal state encoding mechanism and a state-aware adaptive selective attention fusion mechanism to enhance interactions between visual weather information and historical physical patterns. Specifically, the numerical modality captures irradiance trends and temporal correlations through dependency modeling, while the image modality extracts cloud structures and evolution features through spatiotemporal representation. The proposed fusion mechanism dynamically adjusts modality contributions according to forecasting conditions, enabling effective integration of complementary information. Experimental results demonstrate that the proposed framework outperforms benchmark models in forecasting accuracy and robustness, validating the effectiveness of combining visual cloud information with numerical evolution patterns for ultra-short-term solar irradiance forecasting.</p>