杠杆(统计)
计算机科学
推论
语言理解
蒸馏
语言模型
计算
任务(项目管理)
代表(政治)
人工智能
边缘设备
机器学习
GSM演进的增强数据速率
自然语言处理
算法
工程类
有机化学
化学
法学
系统工程
操作系统
政治
云计算
政治学
作者
Victor Sanh,Lysandre Debut,Julien Chaumond,Alexander M. Rush
出处
期刊:Cornell University - arXiv
日期:2019-01-01
被引量:3817
标识
DOI:10.48550/arxiv.1910.01108
摘要
As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive biases learned by larger models during pre-training, we introduce a triple loss combining language modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device study.
科研通智能强力驱动
Strongly Powered by AbleSci AI