Hard template-directed growth methods present a compelling route for the synthesis of Ag nanostructures with precise size control. Meanwhile, soft template methods are effective and flexible for the synthesis of Ag nanostructures with various morphologies. However, the role of the soft template is ambiguous and obviously neglected in hard template-directed growth processes due to the strong confinement effect of the hard template, limiting the diversity of Ag nanostructures that can be obtained. Herein, we design Au nanoframes with deformable head structures as a hard template while using cetyltrimethylammonium chloride as a soft template, to direct the growth of Ag atoms on Au nanobipyramid seeds. When using the Au nanoframes with a closed head, the longitudinal growth of the Ag atoms is clearly limited by the hard template, leading to the formation of thick Ag nanorods with a five-fold twinned structure. The soft template starts to influence the growth process when the head structure of the Au nanoframes becomes hollow. In particular, the confinement effect of the hard template can be completely broken by selectively strengthening the role of the soft template, promoting the production of slender Ag nanorods similar to the results obtained in the absence of the hard template. Our results indicate that the morphology of the Ag nanostructures depends on the competition between the qualitatively confined energies of the hard and soft templates during the template-directed growth process. Moreover, this confined growth mechanism is also verified by the successful construction of various Ag nanostructures. The understanding of the collaborative competition mechanism between the soft and hard templates presents a great opportunity to construct novel Ag nanostructures through a template-directed method.