Abstract
<title>Abstract</title> <p>Purpose Oral squamous cell carcinoma ranks as the most prevalent head and neck cancer globally, with a persistently low 5-year survival rate despite advancements in treatment. Accurately diagnosing oral squamous cell carcinoma and its histopathological grades remains crucial yet challenging due to inter-pathologist variability. This study leverages state-of-the-art deep learning models to enhance the precision and effectiveness of the microscopic image classification of oral squamous cell carcinoma. Method The study utilized 750 microscopic histological images, including normal oral mucosa, low-grade, and high-grade oral squamous cell carcinoma. The Synthetic Minority Over-Sampling Technique and data augmentation methods addressed class imbalance. Pre-trained models, including ResNet50, InceptionV3, VGG16, EfficientNetB7, DenseNet121, Xception, and MobileNet, were fine-tuned through transfer learning. Hyperparameters were optimized using K-fold cross-validation and freezing strategies, allowing model layers to adapt during training. Results EfficientNetB7 emerged as the best performer, achieving an average accuracy and F1-score of 99.2% with 15 layers unfrozen. ResNet50 closely followed with a 99.04% accuracy. Meanwhile, models like VGG16 and MobileNet achieved slightly lower performances, with accuracies of 98.66% and 95.2%, respectively. InceptionV3, despite employing similar techniques, showed the least effective performance at 92.94% accuracy. The findings underscore EfficientNetB7's dominance and ResNet50's reliability for classifying oral squamous cell carcinoma. Conclusion EfficientNetB7 and ResNet50 demonstrated remarkable feasibility in classifying oral squamous cell carcinoma images with high precision, paving the way for enhanced diagnostic tools. This study underscores the potential of transfer learning and advanced deep learning algorithms for improving histopathological cancer diagnoses, especially with a limited sample of microscopic images.</p>