Due to the significant intra-class variability and high inter-class similarity of corals in underwater environments, it is extremely difficult to accurately classify coral image. Hence, a coral image classification approach based on dual-branch feature fusion (DBFF) neural network is proposed, one branch is the residual network to fuse high-dimensional and low-dimensional features, and the other one is multi-scale feature extraction using pyramid convolution. Then, the features of the two branches are fused to obtain richer global features with detailed information to better distinguish similar species. The experimentation result on the coral dataset StructureRSMAS shows that classification accuracy of DBFF is 92.40%, which achieves a higher classification accuracy than that of the existing methods.