Forecasting the trajectory and intensity of tropical cyclones (TCs) is important in disaster mitigation as TC usually causes huge damages. However, it remains a substantial challenge due to the limited understanding of TC complexity. Still, TCs have been observed and recorded for several decades, and so can be predicted if viewed as a spatial-temporal prediction problem with a huge amount of existing data. We propose a novel TC trajectory and intensity short-term prediction method: Multi-Modal Spatial-temporal Networks (MMSTN). It not only predicts the TC’s central pressure, winds, and the location of its center, but also forecasts the TC’s varied possible tendencies. Experiments were conducted on the China Meteorological Administration Tropical Cyclone Best Track Dataset. Experimental results show that the proposed MMSTN outperformed state-of-the-art methods as well as the official prediction method of the China Central Meteorological Observatory (CMO), in intensity prediction and 6 h trajectory prediction.