Accurate network modeling can be used to help optimize load balancing or routing/flow scheduling strategies to ensure Quality of Service (QoS). However, existing network modeling methods have some disadvantages, such as, not being suitable for actual networks and low generalization. This article proposes a Link Delay Model (LDM) based on graph neural network (GNN). The key idea is inspired from the following observations: there is an inherent correlation between the delay, jitter, packet loss, and throughput of each link (this article calls them the basic network behavior), and the basic network behaviors of some links can fully decide and reflect the global network behavior (e.g., end-to-end delay). Firstly, this article proposes two link selection schemes (i.e., all links and few common links selected by clustering). Then, we use an improved GNN to learn the inherent relationship between the basic network behaviors of selected links and the global network behavior. Where the improved GNN uses multiple RNN iterations to aggregate messages in the message aggregation stage. The experiment results verify the feasibility and effectiveness of LDM. When using all links, LDM can accurately predict the end-to-end delay (R2=0.969). Compared with Queuing model and RouteNet, R2 is increased by 73% and 11%, respectively; under unknown flow scheduling strategy, the generalization ability of LDM (MRE=0.285) is also much better than Queuing model and RouteNet. When using partial common links, LDM has close prediction to RouteNet but reduces overhead by 78%.