Beyond Songs: Analyzing User Sentiment Through Music Playlists and Multimodal Data

The automatic recognition of user sentiments through their music listening behavior is an important research task in cognitive studies. Whereas prior studies were conducted to identify the sentiment conveyed (or evoked) by a song that a user listens to at a particular time, we argue that a more effective method would be to identify the user’s induced sentiment based on the comprehensive list of songs they have listened to (e.g., the sequence of music being played). However, recognizing the sentiment information induced by a playlist using machine learning techniques is much more challenging than identifying the sentiment induced by a single song, as it is difficult to obtain accurately labeled training samples for playlists. In this study, we developed the List–Song Relationship Factorization (LSRF) model with the objective of efficiently identifying sentiments induced by playlists. This model employs two side information constraints: the sentiment similarity between songs, based on multimodal information, and the co-occurrence of songs in playlists. These constraints enable the simultaneous co-clustering of songs and playlists. The experimental results demonstrate that the proposed model efficiently and consistently identifies sentiment information evoked by either playlists or individual songs.