Infrared division-of-time (DoT) polarization imaging is a common polarization imaging mode, and it is usually necessary to use an infrared polarized radiation source composed of a blackbody and a polarizer to provide polarized radiation incidence and realize system calibration. The traditional visible-light polarized radiation source only needs to consider the energy attenuation caused by the transmittance of polarizer, that is, the emitted radiation is completely polarized radiation. However, in the thermal infrared band, it is necessary to consider the nonpolarized spontaneous radiation and polarized reflected radiation of the polarizer, which will bias the ideal polarized radiation estimation and affect the calibration accuracy of the infrared DoT polarization imaging system. In this paper, the influence of non-ideal infrared polarized radiation source composed of blackbody and polarizer on the calibration of infrared DoT polarization imaging system is studied, and a calibration model of infrared DoT polarization imaging system is proposed. The experimental results show that the modified model can improve the calibration accuracy of key parameters of the system, restrain the over-calibration, and obtain stable polarization imaging effect.