Abstract
<jats:p>Introduction. Cervical cancer remains one of the leading causes of cancer-related mortality among women worldwide, highlighting the need for effective automated methods for cytological diagnostics. Objective. To develop and validate a deep learning model for the automated segmentation and classification of cells exhibiting signs of malignant transformation in cervical cytological specimens. Materials and Methods. The study included microscopic images of 124 cervical smear specimens obtained from 87 patients aged 21–65 years at the laboratory of the Educational and Scientific Medical Centre “University Clinic” of Zaporizhzhia State Medical University. A total of 18,340 cells were annotated. Cell segmentation was performed using a U-Net architecture with a ResNet-50 encoder, whereas multiclass classification was conducted using a ResNet-50–based convolutional neural network with a modified classification block. Model training was implemented in the PyTorch framework (version 2.5.1) using the Adam optimizer, targeted data augmentation, and a class-weighted loss function to compensate for class imbalance. Model interpretability was assessed using the Grad-CAM technique. Results. The classification model achieved a global accuracy of 0.97 ± 0.03, specificity of 0.96 ± 0.07, sensitivity of 0.93 ± 0.04, precision of 0.87 ± 0.02, and an F1-score of 0.91 ± 0.03. Segmentation performance for cell contours demonstrated a mean Dice coefficient of 0.847 ± 0.031 and a mean intersection-over-union (IoU) score of 0.736 ± 0.028. Nucleus segmentation yielded a mean Dice coefficient of 0.821 ± 0.035 and a mean IoU of 0.711 ± 0.033. Conclusion. The developed deep learning model demonstrated high performance in the automated detection of atypical cells in cervical cytological smears and may serve as a promising decision-support tool in the cytological diagnosis of cervical cancer.</jats:p>