Sodium-ion batteries (SIBs), as one of the most promising energy storage systems, have attracted extensive attention due to abundant sodium resource and low cost. Among various anode materials for SIBs, hard carbon has received more and more attention because of low cost, renewable resources and high capacity. Up to now, many researches have been done to investigate the preparation, microstructure, electrochemical performance and sodium storage mechanism of hard carbon. However, the initial Coulombic efficiency (ICE) of hard carbons is still low and the sodium storage mechanism is controversial because of hard carbons having complex and diverse microstructure, which limit their practical applications and hinder the structural design and preparation of high-performance hard carbons. This review provides a complete overview of the microstructure, sodium storage mechanism and ICE of hard carbons, and highlights the optimization strategies for improving the ICE of hard carbons, including microstructure optimization, surface modification, electrolyte optimization, binder selection and pre-sodiation. Furthermore, the challenges and perspectives in developing hard carbon anodes with high ICE are also discussed in this review, which will be beneficial to development and commercialization of SIBs with high energy density.