Estimating the degree of conflict in speech by employing Bag-of-Audio-Words and Fisher Vectors

The automatic detection of conflict situations from human speech has several straightforward applications such as the surveillance of public spaces, providing feedback about employees in call centers, and other roles in human–computer interactions. In this study we examine the potential of different state-of-the-art feature extraction techniques, all developed to be able to efficiently represent a variable-length speech utterance by a fixed-length feature vector. Besides the ‘ComParE functionals’ attribute set, which became the de facto standard feature set in the area of computational paralinguistics (which focuses on the automatic assessment of non-verbal phenomena being present in human speech), we experiment with two methods introduced quite recently: Bag-of-Audio-Words (BoAW) and Fisher Vectors (FV). Using three standard basic, low-level feature sets, we found that, while BoAW proved to be quite sensitive to its meta-parameter settings, with Fisher Vectors we were able to achieve state-of-the-art conflict intensity estimation performance on a public and widely-used corpus. Furthermore, by applying Principal Component Analysis on the frame-level attributes, we managed to achieve a 30% speed-up in the feature extraction step. Interestingly, in contrast with our previous paralinguistic studies, combining the different predictions with these feature extraction approaches, we were unable to achieve any further significant improvement. The highest correlation coefficient values we got on the test set lay in the range 0.850–0.860, while the authors of several previous studies were able to achieve similar values (i.e. 0.849, 0.856 and 0.853). Considering that in this task the target score to be estimated (i.e. the intensity of the conflict being present in the actual clip) is definitely prone to subjectivity and therefore to label noise, current efforts have probably achieved the highest correlation coefficients attainable, and match human performance.