We have witnessed great success of Single Image Super-Resolution (SISR) with convolutional neural networks (CNNs) in recent years. However, most existing Super-Resolution (SR) networks fail to utilize the multi-scale features of low-resolution (LR) images to further improve the representation capability for more accurate SR. In addition, most of them do not exploit the hierarchical features across networks for the final reconstruction. In this paper, we propose a novel multi-scale feature fusion residual network (MSFFRN) to fully exploit image features for SISR. Based on the residual learning, we propose a multi-scale feature fusion residual block (MSFFRB) with multiple intertwined paths to adaptively detect and fuse image features at different scales. Furthermore, the outputs of each MSFFRB and the shallow features are used as the hierarchical features for global feature fusion. Finally, we recover the high-resolution image based on the fused global features. Extensive experiments on four standard benchmarks demonstrate that our MSFFRN achieves better accuracy and visually pleasing than the current state-of-the-art methods.