Extracting keyframes of breast ultrasound video using deep reinforcement learning

Ultrasound (US) plays a vital role in breast cancer screening, especially for women with dense breasts. Common practice requires a sonographer to recognize key diagnostic features of a lesion and record a single or several representative frames during the dynamic scanning before performing the diagnosis. However, existing computer-aided diagnosis tools often focus on the final diagnosis process while neglecting the influence of the keyframe selection. Moreover, the lesions could have highly-irregular shapes, varying sizes, and locations during the scanning. The recognition of diagnostic characteristics associated with the lesions is challenging and also faces severe class imbalance. To address these, we proposed a reinforcement learning-based framework that can automatically extract keyframes from breast US videos of unfixed length. It is equipped with a detection-based nodule filtering module and a novel reward mechanism that can integrate anatomical and diagnostic features of the lesions into keyframe searching. A simple yet effective loss function was also designed to alleviate the class imbalance issue. Extensive experiments illustrate that the proposed framework can benefit from both innovations and is able to generate representative keyframe sequences in various screening conditions.