HoogBERTa: Multi-task Sequence Labeling using Thai Pretrained Language Representation

Recently, pretrained language representations like BERT and RoBERTa have drawn more and more attention in NLP. In this work we propose a pretrained language representation for Thai language, which based on RoBERTa architecture. Our monolingual data used in the training are collected from publicly available resources including Wikipedia, OpenSubtitles, news and articles. Although the pretrained model can be fine-tuned for wide area of individual tasks, fine-tuning the model with multiple objectives also yields a surprisingly effective model. We evaluated the performance of our multi-task model on part-of-speech tagging, named entity recognition and clause boundary prediction. Our model achieves the comparable performance to strong single-task baselines. Our code and models are available at https://github.com/lstnlp/hoogberta.