Automatic Keyframe Detection for Critical Actions from the Experience of Expert Surgeons

Robot-Assisted Minimally Invasive Surgery (RAMIS), which introduced robot-actuated invasive tools to increase the dexterity and efficiency of traditional MIS, has become popular. Investigations on how to achieve autonomy in RAMIS have drawn vast intention recently, which urges further insights into the process of the surgical procedures. In this paper, the definition of critical actions, which discriminates the essential stages from regular surgical actions, is proposed to help decompose the complicated surgical processes. A critical intra-operative moment of the surgical workflow, which is called the keyframe, is introduced to indicate the beginning or ending moments of the critical actions. A keyframe detection method is proposed for critical action identification based on a new in-vivo dataset labeled by expert surgeons. Surgeons' criteria for critical actions are captured by the explainable features, which can be extracted from the raw laparoscopic images with a two-stage network. Motivated by the surgeon's decision process of keyframes, a hierarchical structure is designed for keyframe identification by checking the spatial-temporal characteristics of the explainable features. Experimental results show that the reliability of the proposed method for keyframe detection achieves unanimous agreement by expert surgeons.