The study of electronic properties of charged defects plays a crucial role in advancing our understanding of how defects influence conductivity, magnetism, and optical behavior in various materials. However, despite its significance, research on large-scale defective systems has been hindered by the high computational cost associated with density functional theory (DFT). In this study, we propose HamGNN-Q, an E(3) equivariant graph neural network framework capable of accurately predicting DFT Hamiltonian matrices for diverse point defects with varying charges, utilizing a unified set of network weights. By incorporating background charge features into the element representation, HamGNN-Q facilitates a direct mapping from structural characteristics and background charges to the electronic Hamiltonian matrix of charged defect systems, obviating the need for DFT calculations. We showcase the model's high precision and transferability by evaluating its performance on GaAs systems encompassing diverse charged defect configurations. Furthermore, we predicted the wave function distribution of polarons induced by defects. We analyzed the node features through principal component analysis, providing physical insights for the interpretability of the HamGNN-Q model. Our approach provides a practical solution for accelerating electronic structure calculations of neutral and charged defects and advancing the design of materials with tailored electronic properties.