Trajectory prediction is imperative in the operation of autonomous vehicles because it aids in understanding the surrounding environment through perception fusion of multiple sensors and high-accuracy localisation. Furthermore, the output of algorithms used for trajectory prediction is used in decision making and path planning. However, the accuracy of trajectory prediction is dependent on the perception system particularly when low-cost sensors are used or severe weather conditions are encountered. Data-dependent and task-dependent aleatoric uncertainty affects the trajectory prediction task. Therefore, in this study, we proposed a network for trajectory prediction that integrates the temporal pattern attention and the graph convolutional sequence encoding to address the data-dependent uncertainty caused by observation noise and machine-perceptual errors. Furthermore, an adaptive weight loss function was developed to address the training difficulties caused by loss-dominated tasks (task-dependent aleatoric uncertainty) in a trajectory prediction network. The results of our experiment indicate a 20% improvement in predictive performance over existing methods under a 5-s prediction horizon. Furthermore, an experiment comparing robustness to heteroscedastic uncertainty confirmed that the proposed method improves trajectory prediction robustness against observation noise and perceptual errors.