Efficient automatic 3D segmentation of cell nuclei for high-content screening

Abstract Background High-content screening (HCS) is a pre-clinical approach for the assessment of drug efficacy. On modern platforms, it involves fluorescent image capture using three-dimensional (3D) scanning microscopy. Segmentation of cell nuclei in 3D images is an essential prerequisite to quantify captured fluorescence in cells for screening. However, this segmentation is challenging due to variabilities in cell confluency, drug-induced alterations in cell morphology, and gradual degradation of fluorescence with the depth of scanning. Despite advances in algorithms for segmenting nuclei for HCS, robust 3D methods that are insensitive to these conditions are still lacking. Results We have developed an algorithm which first generates a 3D nuclear mask in the original images. Next, an iterative 3D marker-controlled watershed segmentation is applied to downsized images to segment adjacent nuclei under the mask. In the last step, borders of segmented nuclei are adjusted in the original images based on local nucleus and background intensities. The method was developed using a set of 10 3D images. Extensive tests on a separate set of 27 3D images containing 2,367 nuclei demonstrated that our method, in comparison with 6 reference methods, achieved the highest precision (PR = 0.97), recall (RE = 0.88) and F1-score (F1 = 0.93) of nuclei detection. The Jaccard index (JI = 0.83), which reflects the accuracy of nuclei delineation, was similar to that yielded by all reference approaches. Our method was on average more than twice as fast as the reference method that produced the best results. Additional tests carried out on three stacked 3D images comprising heterogenous nuclei yielded average PR = 0.96, RE = 0.84, F1 = 0.89, and JI = 0.80. Conclusions The high-performance metrics yielded by the proposed approach suggest that it can be used to reliably delineate nuclei in 3D images of monolayered and stacked cells exposed to cytotoxic drugs.