Three-dimensional localization microscopy using deep learning

Single molecule localization microscopy (SMLM) is one of the fastest evolving and most broadly used super-resolving imaging techniques in the biosciences. While image recordings could take up to hours only ten years ago, scientists are now reaching for real-time imaging in order to follow the dynamics of biology. To this end, it is crucial to have data processing strategies available that are capable of handling the vast amounts of data produced by the microscope. In this article, we report on the use of a deep convolutional neural network (CNN) for localizing particles in three dimensions on the basis of single images. In test experiments conducted on fluorescent microbeads, we show that the precision obtained with a CNN can be comparable to that of maximum likelihood estimation (MLE), which is the accepted gold standard. Regarding speed, the CNN performs with about 22k localizations per second more than three orders of magnitude faster than the MLE algorithm of ThunderSTORM. If only five parameters are estimated (3D position, signal and background), our CNN implementation is currently slower than the fastest, recently published GPU-based MLE algorithm. However, in this comparison the CNN catches up with every additional parameter, with only a few percent extra time required per additional dimension. Thus it may become feasible to estimate further variables such as molecule orientation, aberration functions or color. We experimentally demonstrate that jointly estimating Zernike mode magnitudes for aberration modeling can significantly improve the accuracy of the estimates.