The all-fiber photoacoustic system can work properly in the harsh environment of electromagnetic interference and has the potential advantage of achieving a wide range of distributed detection. However, the ultrasound signal of all-fiber system is affected by signal mode mixing, which will have an impact on defect identification and localization, and the time varying filtering based empirical mode decomposition (TVF-EMD) algorithm is used to solve the effect of mode mixing. According to different modes with different signal characteristics, this work proposes a defect identification and localization method based on an all-fiber photoacoustic NDT system. After experimental testing, the method can identify and locate the number and shape of defects in a thin metal plate with a bar-shaped penetration defect of aluminum alloy 6061 of size 50*50*0.1 cm3.