Multi-scale cross-attention transformer via graph embeddings for few-shot molecular property prediction

Molecular property prediction is a critical step in drug discovery. Deep learning (DL) has accelerated the discovery of compounds with desirable molecular properties for successful drug development. However, molecular property prediction is a low-data problem which makes it hard to solve by regular DL approaches. Graph neural networks (GNNs) operate on graph-structured data using neighborhood aggregation to facilitate the prediction of molecular properties. Nonetheless, GNNs struggle to model the global-semantic structure of graph embeddings for molecular property prediction. Recently, Transformer networks have emerged to model such long-range interactions of molecular embeddings at different scales to predict downstream molecular property tasks. Yet, extending this behavior to molecular embeddings and enabling its training on small biological datasets remains an important challenge in drug discovery. In this work, we study how to learn multi-scale representations from comprehensive graph embeddings for molecular property prediction. To this end, we propose a few-shot GNN-Transformer architecture to combine graph embedding tokens of different sizes and produce stronger features for representation learning. A multi-scale Transformer applies a cross-attention mechanism to exchange information of deep representations fused across two separate branches for small and large embeddings. In addition, a two-module meta-learning framework iteratively updates model parameters across tasks to predict new molecular properties on few-shot data. Extensive experiments on multi-property prediction datasets demonstrate the superior performance of the proposed model when compared with other standard graph-based methods. The code and data underlying this article are available in the repository: https://github.com/ltorres97/FS-CrossTR.