In passive sonars, the time delay estimation methods based on phase information are robust and accurate. However, these methods face a serious decrease in the accuracy of time delay estimation when phase wrapping arises. This paper proposes an off-grid time delay estimation (OGTDE) method suitable for passive sonars. This proposed method provides high-precision time delay estimation results, while avoiding the phase wrapping problem. The simulations show that the accuracy of time delay estimation of our proposed method is superior to that of the reference method under the same conditions.