Replacement of solid polymer electrolyte (SPE) to routine liquid electrolyte is highly desirable for high-energy-density lithium sulfur batteries due to its low volatility, high safety and the ability to suppress shuttle effect. Herein, a polymer-in-salt polysiloxane SPE was fabricated with bi-grafted polysiloxane copolymer, lithium bis(trifluoromethanesulfonyl)imide and poly(vinylidene fluoride), which shows higher ionic conductivity (7.8 × 10-4 S cm-1 at 25 °C). To obtain the satisfactory ionic conductivity and high mechanical property of solid electrolyte simultaneously, cellulose acetate matrix was combined as a rigid substrate to prepare composite polymer electrolyte (CPE) that possesses high ionic conductivity (4.0 × 10-4 S cm-1), enhanced mechanical strength (6.8 MPa), wide electrochemical stability window (4.7 V vs. Li+/Li), and high ion transference number (0.52) at ambient temperature. In addition, the CPE effectively inhibits the growth of lithium dendrites and diffusion of polysulfides. The assembled lithium sulfur battery with CPE exhibits good room-temperature cycling performance at 1C, which indicates that such polymer-in-salt polysiloxane based composite electrolyte membranes can be applied to ambient temperature all-solid-state lithium sulfur batteries.