A dual-strategy residual feed detection algorithm based on improved You Only Look Once (YOLO) v3 and Mask Region-Based Convolutional Networks (RCNN) was explored for detecting residual feed in aquaculture areas. After improving the YOLO v3 network structure, the program can better extract the boundary box of swimming crabs and recognize tiny targets. Using the customized Mask RCNN, the region proposal network (RPN) traversed the fixed-size prior frame for small targets, which reduced false negatives in the inference process and speeded up the network. Using the YOLO v3 output, which extracts image information for swimming crab, as the input for Mask RCNN, the RPN recommended region and the probability of background misidentification were further reduced, which made the improved network more rapid and accurate. The precision of the local rapid detection model was 84%, the real-time detection time was 0.05 s, the detection precision was 96.5%, and the recall rate was 97%. Thus, the dual-strategy detection algorithm for residual feed developed herein could accurately identify residual feed and effectively reduce feed waste.