Noble metal is usually used to improve the gas sensing performance of metal oxide semiconductor (MOS) due to its better catalytic properties. In this work, we reported a synthesis of Pd/ZnO nanocomposite by an in situ reduction with ascorbic acid (AA). It was found that Pd/ZnO sensor has excellent selectivity to CO and the response of the Pd/ZnO sensor towards 100 ppm CO was as high as 15 (Ra/Rg), obviously higher than that of the pristine ZnO sensor (1.4) when the working temperature is 220 °C. Moreover, the pure ZnO sensor almost has no selectivity to CO, but the Pd/ZnO sensor has excellent selectivity to CO, which may be ascribed to the electronic sensitization of Pd. Our present results demonstrate that the Pd can significantly improve the gas-sensing performance of metal oxide semiconductor and the obtained sensor has great potential in monitoring coal mine gas.