In this study, we systematically modified the polymer structure by incorporating non-conjugated units and substituting atoms in the fused-ring core, resulting in significant alterations to intermolecular interaction energies of −974.1, −1076.4, and −1115.1 kJ/mol between the polymer:chloroform and polymer:ethylene glycol. These modifications greatly enhance the orientation of the all-polymer backbone and the intrinsic polarization-sensitive performance of the device, demonstrating exceptional capabilities in polarized light detection and imaging. The optimal device exhibits an ultra-fast response speed of 10.6 μs, a wide frequency response of 17.78 kHz, and a large linear dynamic range of 132.3 dB with good imaging ability, indicating potential benefits for a diverse array of extreme applications. This work illustrates how the strategy of incorporating non-conjugated units and modifying the core atoms can contribute to continuous improvements in molecular orientation and polarized light detection.