Abstract It is universally acknowledged that highly photosensitive transistors are strongly dependent on the high carrier mobility of polymer‐based semiconductors. However, the polymer π–π stacking and aggregation, required to increase the charge mobility, conversely inhibit the dissociation of photogenerated charge carriers, in turn accelerating the geminate recombination of electron‐hole pairs. To explore the effects of charge mobility and polymer stacking on the photoresponsivity of the phototransistors, here, two alternating copolymers are synthesized, namely P‐PPAB‐IDT and P‐PPAB‐BDT, by palladium‐catalyzed Stille coupling of PPAB with indaceodithiophene (IDT) or benzo[1,2‐b:4,5‐b′]dithiophene‐2,6‐diyl) (BDT) monomers. The polymer P‐PPAB‐IDT demonstrates a nearly 20 times enhancement in the hole mobility compared to P‐PPAB‐BDT. Yet, P‐PPAB‐IDT surprisingly shows no response to white light illumination, whereas P‐PPAB‐BDT exhibits a significant photoresponse to the same light source with a high light‐current/dark‐current ( I light / I dark ) ratio of 21.6 in the p‐type area and a low current ratio of just 5.2 in the n‐type area. It is believed that this work will provide an effective strategy to develop highly photosensitive polymer semiconductors by reducing polymer stacking and aggregation rather than improving the charge carrier mobility.