Abstract
<title>Abstract</title> <p>Continuously evolving DeepFake techniques threaten visual media authenticity, thus making Face Forgery Detection (FFD) essential for trustworthy digital media. In the literature, despite existing detectors achieving promising intra-dataset performance, they often degrade substantially on unseen DeepFake methods or datasets, which has motivated recent Contrastive Language-Image Pre-training (CLIP)-based detectors to leverage transferable vision-language semantics for more generalizable forgery detection. Nevertheless, such semantic transferability does not ensure reliable forgery perception, since real and fake faces share similar high-level semantics while discriminative cues are often sparse, local, and fine-grained, making it challenging to preserve transferable semantics and capture localized artifacts simultaneously. To address this issue, we therefore propose a Semantic-Consistent Multi-Granularity Fusion framework (SemFusion), which aims to preserve CLIP's transferable semantic structure while enhancing localized forgery perception. Specifically, we introduce Semantic Consistency Regularization (SCR), which uses the frozen CLIP text space as a semantic reference to regularize the adapted visual representation, further stabilizing CLIP's transferable semantics during visual adaptation. Meanwhile, we develop Multi-level Patch Evidence (MPE) learning to mine suspicious local regions from multi-layer visual patch tokens, whose local evidence is fused with global evidence at the logit level for complementary decision making. Extensive experiments demonstrate that SemFusion outperforms the state-of-the-art methods.</p>