TLS-MHSA: An Efficient Detection Model for Encrypted Malicious Traffic based on Multi-Head Self-Attention Mechanism

In recent years, the use of TLS (Transport Layer Security) protocol to protect communication information has become increasingly popular as users are more aware of network security. However, hackers have also exploited the salient features of the TLS protocol to carry out covert malicious attacks, which threaten the security of network space. Currently, the commonly used traffic detection methods are not always reliable when applied to the problem of encrypted malicious traffic detection due to their limitations. The most significant problem is that these methods do not focus on the key features of encrypted traffic. To address this problem, this study proposes an efficient detection model for encrypted malicious traffic based on transport layer security protocol and a multi-head self-attention mechanism called TLS-MHSA. Firstly, we extract the features of TLS traffic during pre-processing and perform traffic statistics to filter redundant features. Then, we use a multi-head self-attention mechanism to focus on learning key features as well as generate the most important combined features to construct the detection model, thereby detecting the encrypted malicious traffic. Finally, we use a public dataset to verify the effectiveness and efficiency of the TLS-MHSA model, and the experimental results show that the proposed TLS-MHSA model has high precision, recall, F1-measure, AUC-ROC as well as higher stability than seven state-of-the-art detection models.