To enhance the catalytic performance of single-metal-site catalysts (SMSCs), regulating the interaction between the active site and substrate is crucial but challenging. Herein, a series of Rh-based SMSCs (Rh/m-3vPAr3-POLs) were designed and synthesized on P-abundant porous organic polymers (POPs) with different electronegativities of frame phosphine. The Rh–P active sites on various POPs were modified by functional groups (−F, −H, −Me, or −OMe). Both the formation of HRh(CO)2(P)2 active species and the insertion of CO were promoted via the electron-accepting property of fluorine, which endowed Rh/m-3vPAr3-POL-F with the best activity (TOF = 3000 h–1), selectivity (>88.1%), l/b ratio (>6.8), and stability (1000 h) for 1-octene hydroformylation in a fixed-bed reactor. Multiple characterization techniques (extended X-ray absorption fine structure, scanning transmission electron microscopy, in situ Fourier-transform infrared spectroscopy, etc.) and density functional theory calculations were employed to get further insights into the microenvironment and active structure of Rh-based SMSCs. This work offers a promising avenue for designing efficient and stable SMSCs in heterogeneous catalysis.