This study developed a novel, high accurate, and robust framework, termed g-SDDL, for structural damage detection (SDD) directly using vibration data without requiring hand-engineered features. Conventional structural health monitoring approaches require advanced techniques and domain expertise to preprocess vibration signals to achieve highly accurate results, but this may impair the possibility of performing real-time monitoring tasks. Thus, directly using vibration data is one of the research directions that opens a new path towards this ambitious goal, which is also the central subject of this study. For effectively using vibration data, one leverages the graph neural network to capture the inherent spatial correlation of sensor locations and the convolution operation to extract underlying vibration signal patterns. In addition, multiple g-SDDL models can be stacked together for addressing multi-damage scenarios. The proposed approach’s viability is quantitatively demonstrated via three case studies with increasing complexities from a 1D continuous concrete beam to a 2D frame structure and to a experimental database from the literature. High damage detection accuracy of more than 90% was consistently obtained, even for the multi-damage scenarios. Furthermore, the performance and robustness of g-SDDL were investigated through comparison, noise-injection, and parametric studies.