Simultaneous Depth Estimation and Localization for Cell Manipulation Based on Deep Learning

Visual localization, which is a key technology to realize the automation of cell manipulation, has been widely studied. Since the depth of field of the microscope is narrow, the planar localization and depth estimation are usually coupled together. At present, most methods adopt the serial working mode of focusing first and then planar localization, but they usually do not have good real-time performance and stability. In this paper, a simultaneous depth estimation and localization network was developed for cell manipulation. The network takes a focused image and a defocus-offset image as inputs, and outputs the defocus in the depth direction and the offset in the plane at the same time after going through defocus-offset information extraction, defocus classification mapping and offset regression mapping. To train and test our network, we also create two datasets: An Adherent Cell dataset and an Injection Micropipette dataset. The experimental results demonstrated that the proposed method achieves the detection of all test samples with a frame rate of more than 40Hz, and the maximum errors of depth estimation and localization are $\boldsymbol{2.44\mu m}$ and $\boldsymbol{0.49\mu m}$ , respectively. The proposed method has good stability, which is mainly reflected in its strong generalization ability and anti-noise ability.