Retinal vessel segmentation is of great significance for clinical diagnosis of eye-related diseases and diabetic retinopathy. However, due to the imbalance of retinal vessel thickness distribution and the existence of a large number of capillaries, it is difficult to segment the retinal vessels correctly. To better solve this problem, we propose a novel Context Guided Attention Net (CGA-Net) with Joint hard sample mining strategy. Specifically, we propose a Context Guided Attention Module (CGAM) which can utilize both the surrounding context information and spatial attention information to promote the precision of segmentation results. As the CGAM is flexible and lightweight, it can be easily integrated into CNN architecture. To solve the problem of retinal vessel pixel imbalance, we further propose a novel Joint hard sample mining strategy (JHSM) in network training, which combines both the pixel-wise and patch-wise hard mining to largely improve the network's robustness for hard samples. Experiments on publicly DRIVE and CHASE DB1 datasets show that our model outperforms state-of-the-art methods. Our code is available at https://github.com/vignywang/Medical_Seg.