Multi-Source Soft Labeling and Hard Negative Sampling for Retrieval Distractor Ranking

Multiple Choice Questions (MCQs) are a kind of widely adopted approaches in learning assessment. Recently the automatic generation of MCQs has become a popular research area. In this task, Distractor Ranking (DR) is one of the most meaningful and challenging sub-tasks, where the DR models learn to select high-quality distractors from numerous candidates. Currently, some DR methods adopt a two-stage ranking strategy, which brings about a complex process and error propagation. Others directly use Single-Encoder based model to improve the overall performance, which however suffers from low efficiency. To tackle these problems, we propose Retrieval Distractor Ranking (ReDR) task to meet the requirements for practical distractor retrieval scenarios, in which the models should achieve relatively high performance within an acceptable time. In this research, we develop an end-to-end way based on Dual-Encoders framework to solve ReDR task. Besides, we propose multiple kinds of relevance scores including Context-Context, Context-Distractor and Distractor-Distractor, which have been employed in two strategies: 1) Multi-source Soft Labeling (MSL), which assigns each candidate an appropriate soft label from multiple kinds of relevance scores to better simulate the sample distribution of ReDR task; 2) Multi-Source Hard Negative Sampling (MHNS), which selects the hard negative samples according to multiple kinds of relevance scores and further distinguishes the difference between them and the positive samples. The extensive experiments on two well-known MCQ benchmarks have proven the effectiveness of our method.