Multi-stage knowledge distillation for sequential recommendation with interest knowledge

Sequential models based on deep learning are widely used in sequential recommendation task, but the increase of model parameters results in a higher latency in the inference stage, which limits the real-time performance of the model. In order to make the model strike a balance between efficiency and effectiveness, the knowledge distillation technology is adopted to transfer the pre-trained knowledge from the large teacher model to the small student model. We propose a multi-stage knowledge distillation method based on interest knowledge, including interest representation knowledge and interest drift knowledge. In the process of knowledge transfer, expert distillation is designed to transform the knowledge dimension of student model to alleviate the loss of original knowledge information. Specially, curriculum learning is introduced for multi-stage knowledge learning, which further makes the teacher model effectively transfer the knowledge to the student model with limited ability. The proposed method on three real-world datasets including MovieLen-1M, Amazon Game and Steam datasets. The experimental results demonstrate that our method is superior to the other compared distillation method significantly and multi-stage learning makes the student model achieve the knowledge step by step for improvement.