Robust User Behavioral Sequence Representation via Multi-scale Stochastic Distribution Prediction

User behavior representation learned by self-supervised pre-training tasks is widely used in various domains and applications. Conventional methods usually follow the methodology in Natural Language Processing (NLP) to set the pre-training tasks. They either randomly mask some of the behaviors in the sequence and predict the masked ones or predict the next k behaviors. These methods fit for text sequence, in which the tokens are sequentially arranged subject to linguistic criterion. However, the user behavior sequences can be stochastic with noise and randomness. The same paradigm is intractable for learning a robust user behavioral representation.