The rigid-flexible coupling dynamics of a hub-flexible beam system carrying a payload with rotary inertia is presented. Both the transversal deformation and the longitudinal deformation of the flexible beam are considered. And in the total longitudinal deformation the non-linear coupling term, also known as the longitudinal shortening caused by transversal deformation, is considered. The approach of assumed modes is used to describe the deformation of the beam. The rigid-flexible coupling dynamic equations of the system are established via employing the second kind of Lagrange’s equation. Solving for this mathematical model, we can obtain the dynamics of the system. Then the natural frequencies are studied though the transversal bending vibration analysis. The natural frequencies are shown to be proportional to the angular speed ratio and the hub radius ratio, inversely proportional to the payload mass ratio and the rotary inertia ratio. When the angular speed ratio increases up to a certain value, the longitudinal deformation will also have an obvious influence to the transversal bending vibration.