End-to-End Multi-Slice-to-Volume Concurrent Registration and Multimodal Generation

For interventional procedures, a real-time mapping between treatment guidance images and planning data is challenging yet essential for successful therapy implementation. Because of time and machine constraints, it involves imaging of different modalities, resolutions and dimensions, along with severe out-of-plane deformations to handle. In this paper, we introduce MSV-RegSyn-Net, a novel, scalable, deep learning-based framework for concurrent slice-to-volume registration and high-resolution modality transfer synthesis. It consists of an end-to-end pipeline made up of (i) a cycle generative adversarial network for multimodal image translation combined with (ii) a multi-slice-to-volume deformable registration network. The concurrent nature of our approach creates mutual benefit for both tasks: image translation is naturally eased by explicit handling of out-of-plane deformations while registration benefits from bringing multimodal signals into the same domain. Our model is fully unsupervised and does not require any ground-truth deformation or segmentation mask. It obtains superior qualitative and quantitative performance for multi-slice MR to 3D CT pelvic imaging compared to state-of-the-art traditional and learning-based methods on both tasks.