Automated video analysis of emotion and dystonia in epileptic seizures

To investigate the accuracy of deep learning methods applied to seizure video data, in discriminating individual semiologic features of dystonia and emotion in epileptic seizures. A dataset of epileptic seizure videos was used from patients explored with stereo-EEG for focal pharmacoresistant epilepsy. All patients had hyperkinetic (HKN) seizures according to ILAE definition. Presence or absence of (1) dystonia and (2) emotional features in each seizure was documented by an experienced clinician. A deep learning multi-stream model with appearance and skeletal keypoints, face and body information, using graph convolutional neural networks, was used to test discrimination of dystonia and emotion. Classification accuracy was assessed using a leave-one-subject-out analysis. We studied 38 HKN seizure videos in 19 patients. By visual analysis based on ILAE criteria, 9/19 patients were considered to have dystonia and 9/19 patients were considered to have emotional signs. Two patients had both dystonia and emotional signs. Applying the deep learning multistream model, spatiotemporal features of facial appearance showed best accuracy for emotion detection (F1 score 0.84), while skeletal keypoint detection performed best for dystonia (F1 score 0.83). Here, we investigated deep learning of video data for analyzing individual semiologic features of dystonia and emotion in hyperkinetic seizures. Automated classification of individual semiologic features is possible and merits further study. • Deep learning analysis of seizure videos allows automated classification of semiology • Presence of dystonia and/or emotion in hyperkinetic seizures was assessed • Dystonia was best detected by skeletal keypoints, and emotional signs by facial appearance • Spatiotemporal facial features were superior to facial keypoints for emotion detection • Skeletal keypoints topology was superior to spatiotemporal model for dystonia detection