Solid polymer electrolytes with superior flexibility, non-flammability, and good mechanical intensity show great potential in eliminating the flammable hazard of liquid solvent and suppressing lithium dendrite growth. However, inherent limitations (e.g., low ionic conductivity) severely hinder their practical applications. Herein, a novel polymer electrolyte (PHLC), which consists of PVDF-HFP as matrix, LiTFSI as electrolyte filler, and cellulose acetate propionate (CAP) as organic filler, was designed and fabricated by a simple blending process. Since CAP played crucial roles in reducing PVDF-HFP crystallinity, promoting LiTFSI dissociation, and providing migration pathways for Li+, the obtained PHLC (20% CAP) membrane demonstrated an improved ionic conductivity of 1.25 × 10−4 S/cm at 30 °C and a much higher lithium-ion transference number (0.49 vs. 0.29) compared to the membrane without CAP. In addition, the electrochemical stability and cycle stability of membrane were boosted by introducing CAP. After 150 h of Li stripping/plating at 0.1 mA/cm2, the PHLC (20% CAP) membrane only exhibited a tiny polarization voltage of 30 mV. Moreover, the assembled LiFePO4|PHLC|Li cell displayed a reversible capacity of 139.7 mAh/g and a low capacity fading (1.6%) after 70 cycles at 0.2C. This work offers a promising insight into the exploitation of advanced polymer electrolytes.