Automating Prostate Cancer Grading: A Novel Deep Learning Framework for Automatic Prostate Cancer Grade Assessment using Classification and Segmentation

Prostate Cancer (PCa) is the second most common cancer in men and affects more than a million people each year. Grading prostate cancer is based on the Gleason grading system, a subjective and labor-intensive method for evaluating prostate tissue samples. The variability in diagnostic approaches underscores the urgent need for more reliable methods. By integrating deep learning technologies and developing automated systems, diagnostic precision can be improved, and human error minimized. The present work introduces a three-stage framework-based innovative deep-learning system for assessing PCa severity using the PANDA challenge dataset. After a meticulous selection process, 2699 usable cases were narrowed down from the initial 5160 cases after extensive data cleaning. There are three stages in the proposed framework: classification of PCa grades using deep neural networks (DNNs), segmentation of PCa grades, and computation of International Society for Urological Pathology (ISUP) grades using machine learning classifiers. Four classes of patches were classified and segmented (benign, Gleason 3, Gleason 4, and Gleason 5). Patch sampling at different sizes (500 × 500 and 1000 × 1000 pixels) was used to optimize the classification and segmentation processes. The segmentation performance of the proposed network is enhanced by a Self-organized operational neural network (Self-ONN) based DeepLabV3 architecture. Based on these predictions, the distribution percentages of each cancer grade within the whole slide images (WSI) were calculated. These features were then concatenated into machine learning classifiers to predict the final ISUP PCa grade. EfficientNet_b0 achieved the highest F1-score of 83.83% for classification, while DeepLabV3 + architecture based on self-ONN and EfficientNet encoder achieved the highest Dice Similarity Coefficient (DSC) score of 84.9% for segmentation. Using the RandomForest (RF) classifier, the proposed framework achieved a quadratic weighted kappa (QWK) score of 0.9215. Deep learning frameworks are being developed to grade PCa automatically and have shown promising results. In addition, it provides a prospective approach to a prognostic tool that can produce clinically significant results efficiently and reliably. Further investigations are needed to evaluate the framework's adaptability and effectiveness across various clinical scenarios.