Active defect discovery: A human-in-the-loop learning method

AbstractUnsupervised defect detection methods are applied to an unlabeled dataset by producing a ranked list based on defect scores. Unfortunately, many of the top-ranked instances by unsupervised algorithms are not defects, which leads to high false-positive rates. Active Defect Discovery (ADD) is proposed to overcome this deficiency, which sequentially selects instances to get the labeling information (defects or not). However, labeling is often costly. Therefore, balancing detection accuracy and labeling cost is essential. Along this line, this article proposes a novel ADD method to achieve the goal. Our approach is based on the state-of-the-art unsupervised defect detection method, namely, Isolation Forest, as the baseline defect detector to extract features. Thereafter, the sparsity of the extracted features is utilized to adjust the defect detector so that it can focus on more important features for defect detection. To enforce the sparsity of the features and subsequent improvement of the detection accuracy, a new algorithm based on online gradient descent, namely, Sparse Approximated Linear Defect Discovery (SALDD), is proposed with its theoretical Regret analysis. Extensive experiments are conducted on real-world datasets including healthcare, manufacturing, security, etc. The performance demonstrates that the proposed algorithm significantly outperforms the state-of-the-art algorithms for defect detection.Keywords: Isolation forestsparsityactive defect discoverymeasurement feedbackonline gradient descent Data availability statementThe authors confirm that the data supporting the findings of this study are available within the article and its supplementary materials.Notes1 It is actually anomaly detection in an industrial engineering setting.2 The Best-in-class performance that generalizes: https://pyod.readthedocs.io/en/latest/benchmark.html3 http://odds.cs.stonybrook.edu/4 Type I and II errors are not good criteria for our case since Type I error = 1 and Type II error = 0.Additional informationFundingThis project was funded by the Office of Naval Research under Award Number N00014-18-1-2794.Notes on contributorsBo ShenBo Shen is an assistant professor in the Department of Mechanical and Industrial Engineering at the New Jersey Institute of Technology. He received his PhD in industrial and systems engineering at Virginia Tech, Blacksburg, VA, in August 2022. He also received his BS degree in statistics from the University of Science and Technology of China, Hefei, China, in July 2017. His research interests include optimization and machine learning, and data analytics in smart manufacturing, wearable robots, and space weather.Zhenyu (James) KongZhenyu (James) Kong (SM'22) received his BS and MS degrees in mechanical engineering from Harbin Institute of Technology, China, in 1993 and 1995, respectively, and his PhD degree from the Department of Industrial and System Engineering, University of Wisconsin–Madison, Madison, WI, USA, in 2004. He is currently a professor with the Grado Department of Industrial and Systems Engineering, Virginia Tech, Blacksburg, VA, USA. His research interests include sensing and analytics for smart manufacturing, and modeling, synthesis, and diagnosis for large and complex manufacturing systems. He is a fellow of the Institute of Industrial and Systems Engineers (IISE) and the American Society of Mechanical Engineers (ASME). He was recognized as one of the 20 Most Influential Academics in Smart Manufacturing by the Society of Manufacturing Engineering (SME) in 2021.