Generalization of Deep Learning Gesture Classification in Robotic-Assisted Surgical Data: From Dry Lab to Clinical-Like Data

Objective: Robotic-assisted minimally invasive surgery (RAMIS) became a common practice in modern medicine and is widely studied. Surgical procedures require prolonged and complex movements; therefore, classifying surgical gestures could be helpful to characterize surgeon performance. The public release of the JIGSAWS dataset facilitates the development of classification algorithms; however, it is not known how algorithms trained on dry-lab data generalize to real surgical situations. Methods: We trained a Long Short-Term Memory (LSTM) network for the classification of dry lab and clinical-like data into gestures. Results: We show that a network that was trained on the JIGSAWS data does not generalize well to other dry-lab data and to clinical-like data. Using rotation augmentation improves performance on dry-lab tasks, but fails to improve the performance on clinical-like data. However, using the same network architecture, adding the six joint angles of the patient-side manipulators (PSMs) features, and training the network on the clinical-like data together lead to notable improvement in the classification of the clinical-like data. Discussion: Using the JIGSAWS dataset alone is insufficient for training a gesture classification network for clinical data. However, it can be very informative for determining the architecture of the network, and with training on a small sample of clinical data, can lead to acceptable classification performance. Significance: Developing efficient algorithms for gesture classification in clinical surgical data is expected to advance understanding of surgeon sensorimotor control in RAMIS, the automation of surgical skill evaluation, and the automation of surgery.