Antimony selenide (Sb2Se3) has become a potential photovoltaic material due to its attractive feature of high absorption coefficient over a wide range of wavelengths and suitable band gap for effective absorption in the visible regime. However, to efficiently use such materials in photovoltaic and optoelectronic devices, one must be familiar with their carrier and phonon dynamics to understand their mobility and relaxation pathways. In the present work, ultrafast transient reflectance spectroscopy is carried out to observe the transient mechanism associated with external excitation. In this study, we investigated the subsequent relaxation of excited charge carriers in an Sb2Se3 film on various substrates and examined their transitions within the conduction band and intermediate states, probed over a long delay of up to 6 ns. In addition to the influence of substrate-matching conditions on the microstructural change, charge-carrier, and phonon dynamics, the effect of annealing is also investigated. This article reveals the presence of various trap and defect states formed during growth due to lattice mismatching conditions and crystallinity of the deposited films, which can facilitate a wide technological exploration of the development of Sb2Se3 in the field of photovoltaics and optoelectronics.