Evaluating Error Detection Systems

In the first edition of this volume, we painted a gloomy picture of the state-of-the-art in evaluating error detection systems. At that time, unlike other areas of NLP, there was no shared task/repository to establish agreed-upon standards for evaluation. While it is still the case that researchers working in this field often find themselves using proprietary or licensed corpora that cannot be made available to the community as a whole, three shared tasks have now been sponsored so that researchers have the opportunity to compare results on at least some shared training and testing materials. The Helping Our Own (HOO) shared task was piloted in 2011 [Dale and Kilgarriff, 2011a] and was held again in 2012 [Dale et al., 2012]. Grammatical error correction was the featured task at CoNLL 2013 [Ng et al., 2013].