Freshness is the most critical indicator for fruit quality, and directly impacts consumers' physical health and their desire to buy. Also, it is an essential factor of the price in the market. Therefore, it is urgent to study the evaluation method of fruit freshness. Taking banana as an example, in this study, we analyzed the freshness changing process using transfer learning and established the relationship between freshness and storage dates. Features of banana images were automatically extracted using the GoogLeNet model, and then classified by the classifier module. The results show that the model can detect the freshness of banana and the accuracy is 98.92%, which is higher than the human detecting level. In order to study the robustness of the model, we also used this model to detect the changing process of strawberry and found that it is still useful. According to the above results, transfer learning is an accurate, non-destructive, and automated fruit freshness monitoring technique. It may be further applied to the field of vegetable detection.