The concrete compositions have great effects on the drying shrinkage behavior of ultra-high performance concrete (UHPC). Five factors including water-to-binder ratio (w/b) and contents of steel fiber, superplasticizer, silica fume and fly ash were considered to evaluate their influences on the drying shrinkage behaviors of UHPC in this investigation. Based on the drying shrinkage test results, the effect degrees of these factors were analyzed by the gray relation entropy analysis. Then the microstructure and pore distributions of UHPC samples were measured by scanning electron microscope and mercury intrusion porosimetry for mechanism analyses. The results reveal that a lower w/b or higher content of fly ash is conducive to achieving a reduction in drying shrinkage of UHPC, while the additions of superplasticizer and silica fume have an adverse effect on drying shrinkage, resulting in increased drying shrinkage, and the effect of steel fiber on drying shrinkage is not monotonic, its optimum dosage is 50 kg/m3 in this work and a lower or higher amount will cause an increased DSS of UHPC compared with the one with an optimum dosage. The five factors can affect the drying shrinkage behavior of UHPC by principally changing the microtopography and internal pore structure of the concrete matrix, according to the gray relation entropy analysis, the effect degrees of studied factors on drying shrinkage of UHPC are as follows: w/b > superplasticizer > silica fume > fly ash > steel fiber.