Due to the competitive and better performance on various benchmarks of computer vision, transfer learning has been widely explored in the task of crop disease image recognition with small samples. However, when there is a large difference in data distribution between target domain and source domain, it is difficult to achieve good results using transfer learning. To solve this problem, this paper took rice disease image recognition with small samples as an example, and proposed a domain adaptive image recognition method based on a novel attention mechanism. First, the weight distribution of rice disease image features in the neural network was optimized, so that the improved attention mechanism can pay more attention to the image features closely related to rice diseases. Second, the attention mechanism was integrated with the domain adaptation network to reduce the difference of feature distribution between the datasets, thereby improving the accuracy of rice disease image recognition. Finally, some comparative experiments were designed to verify the effectiveness of the proposed method. Experimental results showed that the method can effectively solve the problem of low image recognition accuracy caused by large difference in data distribution between target domain and source domain. And when the size of the source domain dataset decreased gradually, the method can still maintain a high and stable accuracy. The image recognition accuracies of 95.25%, 91.50% and 91.25% were achieved on three commonly used domain adaptation networks respectively. The achievement of this paper can offer a new method for crop disease image recognition with small samples.