When sleeves are grouted at the construction site, grouting defects occur inside the sleeves owing to the structure of the grout sleeve, plugging time of the grout outlet hole, and connection form of the sleeve. Grouting defects reduce the connection strength of grout sleeves and further affect the safety performance of assembled building structures. In this study, the failure modes of grout defects and their influencing factors were fully investigated, which provides guidelines for the standardization of sleeve grouting construction. First, a uniaxial tensile test was performed on 33 grouting defect specimens to study the mechanical properties of the grouting defect. Second, the Concrete damaged plasticity (CDP) and Brittle Cracking (BC) constitutive models of grout were considered and optimized, and the corresponding numerical models were established to further analyze the failure modes of sleeve grouting. The numerical analysis results were compared with the test results. The comparison results show that adopting the optimized CDP constitutive model in the numerical simulation is more consistent with the test results. Finally, the stress mechanism of the grout sleeve was theoretically analyzed, and the calculation equation for the anchorage length was improved. The experimental study revealed that the ultimate bearing capacity and displacement of the specimen were affected by the specimen size, and the large-size specimen was more prone to pull-out failure. In addition, theoretical analysis showed that the specimen was in a safe state when the anchorage length of the rebar was greater than 6d. Based on the investigation results, the improved anchorage length equation can be used to calculate the anchorage length of the grout sleeve in sleeve grouting construction. Moreover, the middle and lateral grout defects of the grout sleeve should be focused on in sleeve grouting construction to ensure the quality of sleeve grouting.