Dynamic differential entropy and brain connectivity features based EEG emotion recognition

International Journal of Intelligent SystemsVolume 37, Issue 12 p. 12511-12533 RESEARCH ARTICLE Dynamic differential entropy and brain connectivity features based EEG emotion recognition Fa Zheng, Fa Zheng orcid.org/0000-0001-9462-5789 School of Information Science and Engineering, Shandong Normal University, Jinan, China Shandong Provincial Key Laboratory for Distributed Computer Software Novel Technology, Jinan, ChinaSearch for more papers by this authorBin Hu, Corresponding Author Bin Hu [email protected] School of Information Science and Engineering, Shandong Normal University, Jinan, China Shandong Provincial Key Laboratory for Distributed Computer Software Novel Technology, Jinan, China Correspondence Bin Hu and Xiaomei Yu, School of Information Science and Engineering, Shandong Normal University, Jinan, China. Email: [email protected] and [email protected]Search for more papers by this authorXiangwei Zheng, Xiangwei Zheng School of Information Science and Engineering, Shandong Normal University, Jinan, China Shandong Provincial Key Laboratory for Distributed Computer Software Novel Technology, Jinan, ChinaSearch for more papers by this authorCun Ji, Cun Ji School of Information Science and Engineering, Shandong Normal University, Jinan, China Shandong Provincial Key Laboratory for Distributed Computer Software Novel Technology, Jinan, ChinaSearch for more papers by this authorJi Bian, Ji Bian School of Information Science and Engineering, Shandong Normal University, Jinan, China Shandong Provincial Key Laboratory for Distributed Computer Software Novel Technology, Jinan, ChinaSearch for more papers by this authorXiaomei Yu, Corresponding Author Xiaomei Yu [email protected] School of Information Science and Engineering, Shandong Normal University, Jinan, China Shandong Provincial Key Laboratory for Distributed Computer Software Novel Technology, Jinan, China Correspondence Bin Hu and Xiaomei Yu, School of Information Science and Engineering, Shandong Normal University, Jinan, China. Email: [email protected] and [email protected]Search for more papers by this author Fa Zheng, Fa Zheng orcid.org/0000-0001-9462-5789 School of Information Science and Engineering, Shandong Normal University, Jinan, China Shandong Provincial Key Laboratory for Distributed Computer Software Novel Technology, Jinan, ChinaSearch for more papers by this authorBin Hu, Corresponding Author Bin Hu [email protected] School of Information Science and Engineering, Shandong Normal University, Jinan, China Shandong Provincial Key Laboratory for Distributed Computer Software Novel Technology, Jinan, China Correspondence Bin Hu and Xiaomei Yu, School of Information Science and Engineering, Shandong Normal University, Jinan, China. Email: [email protected] and [email protected]Search for more papers by this authorXiangwei Zheng, Xiangwei Zheng School of Information Science and Engineering, Shandong Normal University, Jinan, China Shandong Provincial Key Laboratory for Distributed Computer Software Novel Technology, Jinan, ChinaSearch for more papers by this authorCun Ji, Cun Ji School of Information Science and Engineering, Shandong Normal University, Jinan, China Shandong Provincial Key Laboratory for Distributed Computer Software Novel Technology, Jinan, ChinaSearch for more papers by this authorJi Bian, Ji Bian School of Information Science and Engineering, Shandong Normal University, Jinan, China Shandong Provincial Key Laboratory for Distributed Computer Software Novel Technology, Jinan, ChinaSearch for more papers by this authorXiaomei Yu, Corresponding Author Xiaomei Yu [email protected] School of Information Science and Engineering, Shandong Normal University, Jinan, China Shandong Provincial Key Laboratory for Distributed Computer Software Novel Technology, Jinan, China Correspondence Bin Hu and Xiaomei Yu, School of Information Science and Engineering, Shandong Normal University, Jinan, China. Email: [email protected] and [email protected]Search for more papers by this author First published: 02 December 2022 https://doi.org/10.1002/int.23096Read the full textAboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Abstract Emotion recognition has become a research focus in the brain–computer interface and cognitive neuroscience. Electroencephalogram (EEG) is employed for its advantages as accurate, objective, and noninvasive nature. However, many existing research only focus on extracting the time and frequency domain features of the EEG signals while failing to utilize the dynamic temporal changes and the positional relationships between different electrode channels. To fill this gap, we develop the dynamic differential entropy and brain connectivity features based EEG emotion recognition using linear graph convolutional network named DDELGCN. First, the dynamic differential entropy feature which represents the frequency domain feature as well as time domain feature is extracted based on the traditional differential entropy feature. Second, brain connectivity matrices are constructed by calculating the Pearson correlation coefficient, phase-locked value and transfer entropy, and then are used to denote the connectivity features of all electrode combinations. Finally, a linear graph convolutional network is customized and applied to aggregate the features from total electrode combinations and then classifies the emotional states, which consists of five layers, namely, an input layer, two linear graph convolutional layers, a fully connected layer, and a softmax layer. Extensive experiments show that the accuracies in the valence and arousal dimensions reach 90.88% and 91.13%, and the precision reaches 96.66% and 97.02% on the DEAP dataset, respectively. On the SEED dataset, the accuracy and precision reach 91.56% and 97.38%, respectively. CONFLICT OF INTEREST The authors declare no conflict of interest. Open Research DATA AVAILABILITY STATEMENT The two public EEG datasets, namely, DEAP and SEED are used in our paper. We declare both EEG datasets as open access, where the DEAP dataset can be downloaded at http://www.eecs.qmul.ac.uk/mmv/datasets/deap/download.html and the SEED dataset can be downloaded at https://bcmi.sjtu.edu.cn/home/seed/seed.html. REFERENCES 1Zhang J, Yin Z, Chen P, Nichele S. Emotion recognition using multi-modal data and machine learning techniques: a tutorial and review. Inform Fusion. 2020; 59: 103- 126. doi:10.1016/j.inffus.2020.01.011 2Li W, Tan J, Meng W, Wang Y. A swipe-based unlocking mechanism with supervised learning on smartphones: design and evaluation. J Network Comput Appl. 2020; 165:102687. doi:10.1016/j.jnca.2020.102687 3Zhang Y, Nam CS, Zhou G, Jin J, Wang X, Cichocki A. Temporally constrained sparse group spatial patterns for motor imagery BCI. IEEE Trans Cybernet. 2019; 49(9): 3322- 3332. doi:10.1109/TCYB.2018.2841847 4Britton JC, Phan KL, Taylor SF, Welsh RC, Berridge KC, Liberzon I. Neural correlates of social and nonsocial emotions: an fMRI study. NeuroImage. 2006; 31(1): 397- 409. doi:10.1016/j.neuroimage.2005.11.027 5Zhang X, Wang Y, Geng G, Yu J. Delay-optimized multicast tree packing in software-defined networks. IEEE Trans Serv Comput. Published online August 20, 2021. doi:10.1109/TSC.2021.3106264 6Zheng X, Liu X, Zhang Y, Cui L, Yu X. A portable HCI system-oriented EEG feature extraction and channel selection for emotion recognition. Int J Intell Syst. 2021; 36: 152- 176. doi:10.1002/int.22295 7Moon SE, Chen CJ, Hsieh CJ, Wang JL, Lee JS. Emotional EEG classification using connectivity features and convolutional. Neural Networks 2021; 37: 96- 107. doi:10.48550/arXiv.2101.07069 8Cai J, Wang Y, Liu Y, Luo JZ, Wei W, Xu X. Enhancing network capacity by weakening community structure in scale-free network. Future Generation Comput Syst. 2018; 87: 765- 771. doi:10.1016/j.future.2017.08.014 9Wang Y, Meng W, Li W, Li J, Liu WX, Xiang Y. A fog-based privacy-preserving approach for distributed signature-based intrusion detection. J Parallel Distrib Comput. 2018; 122: 26- 35. doi:10.1016/j.jpdc.2018.07.013 10Li W, Ke L, Meng W, Han J. An empirical study of supervised email classification in Internet of things: practical performance and key influencing factors. Int J Intell Syst. 2021; 37: 287- 304. doi:10.1002/int.22625 11Wang H, Li S, Song L, Cui L. A novel convolutional neural network based fault recognition method via image fusion of multi-vibration-signals. Comput Industry. 2019; 105: 182- 190. doi:10.1016/j.compind.2018.12.013 12Scarselli F, Gori M, Tsoi AC, Hagenbuchner M, Monfardini G. The graph neural network model. IEEE Trans Neural Networks. 2009; 20(1): 61- 80. doi:10.1109/TNN.2008.2005605 13Defferrard M, Bresson X, Vandergheynst P. Convolutional neural networks on graphs with fast localized spectral filtering. Proceedings of the 30th International Conference on Neural Information Processing Systems. Curran Associates Inc. 2016: 3844- 3852. 14Wu LW. Enhancing intrusion detection with feature selection and neural network. Int J Intell Syst. 2021; 36: 3087- 3105. doi:10.1002/int.22397 15Takahashi K, Tsukaguchi A. Remarks on emotion recognition from multi-modal bio-potential signals. Proceedings of the IEEE International Conference on Systems; 2003: 1654- 1659. 16Hjorth B. EEG analysis based on time domain properties. Electroencephalogr Clin Neurophysiol. 1970; 29: 306- 10. doi:10.1016/0013-4694(70)90143-4 17Petrantonakis P, Hadjileontiadis L. Emotion recognition from EEG using higher order crossings. IEEE Trans Inform Technol Biomed. 2009; 14: 186- 197. doi:10.1109/TITB.2009.2034649 18Anantharam V. A variational characterization of rnyi divergences. IEEE Trans Inform Theory. 2018; 64: 893- 897. doi:10.1109/TIT.2018.2861013 19Shi LC, Jiao YY, Lu BL. Differential entropy feature for EEG-based vigilance estimation. 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 2013; 2013: 6627- 6630. doi:10.1109/EMBC.2013.6611075 20Ko K, Yang HC, Sim KB. Emotion recognition using EEG signals with relative power values and Bayesian network. Int J Control Autom Syst. 2009; 7: 865- 870. doi:10.1007/s12555-009-0521-0 21Alsolamy M, Fattouh A. Emotion estimation from EEG signals during listening to Quran using PSD features. International Conference on Computer Science and Information Technology; 2016: 1- 5. 22Liu YH, Wu CT, Kao YH, Chen YT. Single-trial EEG-based emotion recognition using kernel Eigen-emotionEigen-emotion pattern and adaptive support vector machine. 2013 International Conference of the IEEE Engineering in Medicine and Biology Society. 2013:4306-4309. 23Jatupaiboon N, Pan-ngum S, Israsena P. Emotion classification using minimal EEG channels and frequency bands. The 2013 10th International Joint Conference on Computer Science and Software Engineering (JCSSE); 2013: 21- 24. 24Jian W, Chen M, McFarland DJ. EEG based zero-phase phase-locking value (PLV) and effects of spatial filtering during actual movement. Brain Res Bull. 2017; 130: 156- 164. doi:10.1016/j.brainresbull.2017.01.023 25Lee YY, Hsieh S. Classifying different emotional states by means of EEG-Based functional connectivity patterns. PloS One. 2014; 9:e95415. doi:10.1371/journal.pone.0095415 26Kouti M, Ansari-Asl K, Namjoo E. Emotion discrimination using source connectivity analysis based on dynamic ROI identification. Biomed Signal Process Control. 2022; 72:103332. doi:10.1016/j.bspc.2021.103332 27Zheng X, Yu X, Yin Y, Li T, Yan X. Three-dimensional feature maps and convolutional neural network-basedemotion recognition. Int J Intell Syst. 2021; 36: 6312- 6336. doi:10.1002/int.v36.11 28Li X, Song D, Zhang P, Yu G, Hou Y, Hu B. Emotion recognition from multi-channel EEG data through convolutional recurrent neural network. 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM); 2016:352-359. 29Yang Y, Wu Q, Qiu M, Wang Y, Chen X. Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network. 2018 International Joint Conference on Neural Networks (IJCNN); 2018:1-7. 30Zheng WL, Lu BL. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Autonom Mental Develop. 2015; 7(3): 162- 175. doi:10.1109/TAMD.2015.2431497 31Tao W, Li C, Song R, et al. EEG-based emotion recognition via channel-wise attention and self attention. IEEE Trans Affect Comput. Published online September 22, 2020. doi:10.1109/TAFFC.2020.3025777 32Zhang T, Su G, Qing C, Xu X, Cai B, Xing X. Hierarchical lifelong learning by sharing representations and integrating hypothesis. IEEE Trans Syst Man, Cybernet Syst. 2021; 51(2): 1004- 1014. doi:10.1109/TSMC.2018.2884996 33Song T, Zheng W, Song P, Cui Z. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput. 2020; 11(3): 532- 541. doi:10.1109/TAFFC.2018.2817622 34Wang Xh, Zhang T, Xu Xm, Chen L, Xing Xf, Chen CLP. EEG emotion recognition using dynamical graph convolutional neural networks and broad learning system. 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM); 2018:1240-1244. 35Yin Y, Zheng X, Hu B, Zhang Y, Cui X. EEG emotion recognition using fusion model of graph convolutional neural networks and LSTM. Appl Soft Comput. 2021; 100:106954. doi:10.1016/j.asoc.2020.106954 36Sartipi S, Torkamani-Azar M, Cetin M. EEG emotion recognition via graph-based spatio-temporal attention neural networks. 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC); 2021:571-574. 37Liu S, Wang X, Zhao L, Zhao J, Xin Q, Wang SH. Subject-independent emotion recognition of EEG signals based on dynamic empirical convolutional neural network. IEEE/ACM Trans Computat Biol Bioinform. 2021; 18(5): 1710- 1721. doi:10.1109/TCBB.2020.3018137 38Molla M, Tanaka T, Rutkowski T. Multivariate EMD based approach to EOG artifacts separation from EEG. 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); 2012:653-656. 39Alam SMS, Bhuiyan MIH. Detection of seizure and epilepsy using higher order statistics in the EMD domain. IEEE J Biomed Health Inform. 2013; 17(2): 312- 318. doi:10.1109/JBHI.2012.2237409 40Lachaux J, Rodriguez E, Martinerie J, Varela F. Measuring phase synchrony in brain signals. Human Brain Mapping. 1999; 8: 194- 208. doi:10.1002/(SICI)1097-0193(1999)8:43.0.CO;2-C 41Schreiber T. Measuring information transfer. Phys Rev Lett. 2000; 85: 461- 464. doi:10.1103/PhysRevLett.85.461 42Li Q, Han Z, Wu XM. Deeper insights into graph convolutional networks for semi-supervised learning. Proc AAAI Conf Artif Intell. 2018; 32. doi:10.1609/aaai.v32i1.11604 43Koelstra S, Muhl C, Soleymani M, et al. DEAP: a database for emotion analysis; using physiological signals. IEEE Trans Affective Comput. 2012; 3(1): 18- 31. doi:10.1109/T-AFFC.2011.15 44Wang XW, Nie D, Lu BL. EEG-based emotion recognition using frequency domain features and support vector machines. Proceedings of the 18th International Conference on Neural Information Processing, Volume Part I; 2011:734-743. 45Ayata D, Yaslan Y, Kamayak M. Emotion recognition via random forest and galvanic skin response: comparison of time based feature sets, window sizes and wavelet approaches. 2016 Medical Technologies National Congress (TIPTEKNO); 2016:1-4. 46Zerdem SM, Polat H. Emotion recognition based on EEG features in movie clips with channel selection. Brain Inform; 2017. doi:10.1007/s40708-017-0069-3 47Tong L, Zhao J, Fu W. Emotion recognition and channel selection based on EEG signal. 2018 11th International Conference on Intelligent Computation Technology and Automation (ICICTA); 2018:101-105. 48Wang ZM, Hu SY, Song H. Channel selection method for EEG emotion recognition using normalized mutual information. IEEE Access. 2019; 7: 143303- 143311. doi:10.1109/ACCESS.2019.2944273 49Tang H, Liu W, Zheng WL, Lu BL. Multimodal emotion recognition using deep neural networks. Neural Inform Process. 2017; 10637: 811- 819. doi:10.1007/978-3-319-70093-9_86 50An Y, Xu N, Qu Z. Leveraging spatial-temporal convolutional features for EEG-based emotion recognition. Biomed Signal Process Control. 2021; 69:102743. doi:10.1016/j.bspc.2021.102743 51Liu W, Zheng WL, Lu BL. Emotion recognition using multimodal deep learning. Neural Inform Process. 2016; 9948: 521- 529. doi:10.1007/978-3-319-46672-9_58 52Zhang T, Zheng W, Cui Z, Zong Y, Li Y. Spatial-temporal recurrent neural network for emotion recognition. IEEE Trans Cybernet. 2019; 49(3): 839- 847. doi:10.1109/TCYB.2017.2788081 Volume37, Issue12December 2022Pages 12511-12533 ReferencesRelatedInformation