Protocol state analysis and annotation methodology based on deep learning

Protocol Reverse Engineering (PRE) is of great significance to the research of cyber security and it is helpful to understand protocol specifications. There has been many researches on PRE but most of them need additional manual analysis, which is not available for private and unknown protocols. We propose a protocol state analysis and annotation method, which extracts the feature information of binary-based protocol data through an auto-encoder model. Moreover, density-based clustering algorithm is only used to distinguish protocol types in existing studies, we propose an improved algorithm and apply it to protocol state analysis. Finally, we apply alignment algorithm to get state information and do annotation. We run simulation to verify the effectiveness of proposed method and prove its feasibility in private and unknown protocols. The results of clustering algorithms are compared to show the improvement. Then the application of proposed method is summarized based on the simulation results, which provides a novel idea for the protocol analysis.