Staining, magnification, and algorithmic conditions for highly accurate cell detection and cell classification by deep learning

Abstract Objectives Research into cytodiagnosis has seen an active exploration of cell detection and classification using deep learning models. We aimed to clarify the challenges of magnification, staining methods, and false positives in creating general purpose deep learning–based cytology models. Methods Using 11 types of human cancer cell lines, we prepared Papanicolaou- and May-Grünwald-Giemsa (MGG)–stained specimens. We created deep learning models with different cell types, staining, and magnifications from each cell image using the You Only Look Once, version 8 (YOLOv8) algorithm. Detection and classification rates were calculated to compare the models. Results The classification rates of all the created models were over 95.9%. The highest detection rates of the Papanicolaou and MGG models were 92.3% and 91.3%, respectively. The highest detection rates of the object detection and instance segmentation models, which were 11 cell types with Papanicolaou staining, were 94.6% and 91.7%, respectively. Conclusions We believe that the artificial intelligence technology of YOLOv8 has sufficient performance for applications in screening and cell classification in clinical settings. Conducting research to demonstrate the efficacy of YOLOv8 artificial intelligence technology on clinical specimens is crucial for overcoming the unique challenges associated with cytology.