Due to fast movement of an autonomous system, motion blur is an issue for images with long-exposures, especially for those captured at dim-lighting conditions. In this paper, a simple semi-blind image de-blurring algorithm is designed by using measurements from an inertial measurement unit (IMU) embedded in the autonomous system. A prior of the blur kernel is first derived from an elegant IMU integration model. The image is then de-blurred by using the prior of the blur kernel with a refinement step to further mitigate the drift caused by the inaccurate IMU measurements. Experimental results show the improvement of the proposed algorithm for image de-blurring.