Molecular language models: RNNs or transformer?

Abstract Language models have shown the capacity to learn complex molecular distributions. In the field of molecular generation, they are designed to explore the distribution of molecules, and previous studies have demonstrated their ability to learn molecule sequences. In the early times, recurrent neural networks (RNNs) were widely used for feature extraction from sequence data and have been used for various molecule generation tasks. In recent years, the attention mechanism for sequence data has become popular. It captures the underlying relationships between words and is widely applied to language models. The Transformer-Layer, a model based on a self-attentive mechanism, also shines the same as the RNN-based model. In this research, we investigated the difference between RNNs and the Transformer-Layer to learn a more complex distribution of molecules. For this purpose, we experimented with three different generative tasks: the distributions of molecules with elevated scores of penalized LogP, multimodal distributions of molecules and the largest molecules in PubChem. We evaluated the models on molecular properties, basic metrics, Tanimoto similarity, etc. In addition, we applied two different representations of the molecule, SMILES and SELFIES. The results show that the two language models can learn complex molecular distributions and SMILES-based representation has better performance than SELFIES. The choice between RNNs and the Transformer-Layer needs to be based on the characteristics of dataset. RNNs work better on data focus on local features and decreases with multidistribution data, while the Transformer-Layer is more suitable when meeting molecular with larger weights and focusing on global features.