We propose a new deep plug-and-play (PnP) algorithm for constrained image restoration with guaranteed theoretical convergence. The PnP strategy, which incorporates off-the-shelf Gaussian denoisers into proximal splitting algorithms, has demonstrated outstanding performance in a variety of image restoration tasks. However, theoretically guaranteeing its convergence under reasonable assumptions is extremely challenging. In particular, an existing PnP method based on a primal-dual splitting algorithm (PnP-PDS) faces instability due to the absence of theoretical convergence guarantees, despite its flexibility in solving constrained image restoration without matrix inversion. In this paper, we address this instability issue by establishing theoretical conditions for the convergence of PnP-PDS with deep neural network (DNN)-based denoisers, and then provide its application to constrained image restoration. We also demonstrate experimentally that our convergent PnP-PDS achieves state-of-the-art performance in two image restoration tasks. These findings support the practicality of our theoretical results in terms of both performance and stability.