Abnormal fish detection helps producers adjust breeding strategies in a timely manner to prevent the occurrence of diseases and improve aquaculture efficiency and quality. The rapid development of computer vision technology provides a noninvasive method for abnormal fish detection, which can be used to identify and classify abnormal fish. This paper provides an overview of the research progress of computer vision techniques in detecting abnormal fish over the past two decades. For the first time, the abnormal fish detection task is divided into three external manifestations: abnormal physiological activities, abnormal trajectories, and abnormal surface features of fish. The traditional methods and deep learning methods in computer vision technology are further summarized for their application approaches in these three research types, and the commonly used classical algorithm models in abnormal fish detection are introduced comprehensively. In addition, this paper summarizes several common methods for obtaining public datasets in aquaculture and evaluation indicators of model accuracy, emphasizing two methods for researchers to collect experimental on-site data. Finally, based on the above work, this paper analyzes several challenges in abnormal fish detection, proposes feasible strategies for each challenge, and notes the importance of improving models to effectively integrate and analyze data from multiple platforms. This paper provides some reference value for research on abnormal fish.