Exploring Metamorphic Testing for Fake-News Detection Software: A Case Study

Concerns have been growing over fake news and its impact. Software that can automatically detect fake news is becoming more popular. However, the accuracy and reliability of such fake-news detection software remains questionable, partly due to a lack of testing and verification. Testing this kind of software may face the oracle problem, which refers to difficulty (or inability) of identifying the correctness of the software’s output in a reasonable amount of time. Metamorphic testing (MT) has a record of effectively alleviating the oracle problem, and has been successfully applied to testing fake-news detection software. This paper reports on a study, extending previous work, exploring the use of MT for fake-news detection software. The study includes new metamorphic relations and additional experimental results and analysis. Some alternative MR-generation approaches are also explored. The study targets software where the output is a real/fake news decision, enhancing the applicability of MT to current fake-news detection software. The paper also explores the impact of the prediction accuracy of the fake-news detection software on the MT process. The study demonstrates the validity and applicability of MT to fake-news detection software. The prediction accuracy of the software has a greater impact on MT experiments with greater changes between the source and follow-up inputs, and less dependence on prediction stability. Some possible factors affecting the experimental results are discussed, and directions for future work are provided.