Sugars are fundamental metabolites synthesized in leaves and further delivered to fruit in fruit crops. They not only provide “sweetness” as fruit quality traits, but also function as signaling molecules to modulate the responses of fruit to environmental stimuli. Therefore, the understanding to the molecular basis for sugar metabolism and transport is crucial for improving fruit quality and dissecting responses to abiotic/biotic factors. Here, we provide a review for molecular components involved in sugar metabolism and transport, crosstalk with hormone signaling, and the roles of sugars in responses to abiotic and biotic stresses. Moreover, we also envisage the strategies for optimizing sugar metabolism during fruit quality maintenance.