Transformer with Local-feature Extractor for Relation Extraction

Many pre-trained models recently achieved successful results in many NLP tasks. However, such models with heavy structure often led to problems such as a great amount of training time or too much computational cost. This paper presents a more light-weighted model based on Transformer and convolutional neural networks. The proposed model is able to capture important local information of different length by multi-head convolutional layer and effectively extracts long term dependency by position-enriched self-attention layer. These two structures together formed our Transformer with Local-feature Extractor (TLE) encoder. Comparing with previous CNN, RNN, and attention-based baseline models, our model achieves the best performance on multiple relation extraction datasets.