Image-based Live Tracking and Registration for AR-Guided Liver Surgery Using Hololens2: A Phantom Study

We investigate AR-based tracking and registration of the liver surface for potential surgical applications. Our approach consists of streaming RGBD data from a Hololens2 device, RGBD segmentation using a deep learning model and registering the acquired partial liver surface point cloud with the corresponding virtual liver model. We aim to derive basic requirements for AR-guided liver surgery, thus consider several test cases of partially occluded liver as it would appear in surgical scenarios. To evaluate our approach, we use a 3D-printed phantom with basic texture and rigid structure. Our results show that the visible liver section has a substantial impact of feature extraction and matching, thus the registration process. Test cases, where specific anatomical features are visible, e.g. the right liver lobe, yielded superior outcomes compared to other cases, e.g. only the left liver lobe visible. Moreover, our results showed that large scale Hololens movements during the tracking process affected the registration performance. Our implementation achieved 2-3 frames per second for tracking and registration. We discuss the potential and limitations of utilizing Hololens2 for real-time tracking and registration of the liver surface. To our knowledge this is the first experimental approach for real-time markerless tracking and registration for AR-guided surgery guidance using the Hololens2 sensors only.