The sea wave is an important factor that causes the sea disaster. A rapid and accurate prediction is of great significance for avoiding the sea wave disaster. In this paper, we establish a fast prediction model based on the deep learning, which is named driving field forced convolution long and short memory network (DFF-ConvLSTM). The wind speed predicted and the reanalysis wave fields are taken as inputs (the forcing and initial conditions respectively), and model outputs the predicted wave fields in the future. The model is trained using the historical sea surface wind and wave dataset of North West Pacific. DFF-ConvLSTM grasps the temporal and spatial evolution law of wave fields after training, which can provide high resolution wave prediction on a basin-scale. The prediction parameters include significant wave height, mean period and average wavelength. The 5-day wave height predictions of the Northwest Pacific produced by DFF-ConvLSTM are highly similar to the numerical results, and the root mean square error between DFF-ConvLSTM and altimeter is 0.439 m. The calculating time is just 2.7s realizing on the graphics computing unit, which is 770 times faster than the numerical model.