Abstract Alternating current (AC) losses have an important effect on the stable operation of high-temperature superconductors (HTSs). Researchers have proposed many formulations to estimate AC losses based on finite element methods, such as T−A formulation, H formulation and H−ϕ formulation. In this paper, in order to simulate complicated multi-layer conductor on round core (CORC) cable with transport current, a modified H−ϕ formulation is proposed to calculate AC losses in complicated HTS structures. The formulation, which is easy to implement and has fast computation speed, is verified by comparing the calculation results with the existing formulations and experiments. The electromagnetic behavior of CORC cables and a helical coil wound by a three-layer CORC cable is investigated with a modified H−ϕ formulation. The numerical results show the non-uniformity of the losses and currents in the different layers of the CORC cable. Furthermore, the effects of the external magnetic field on the distribution of the transport current are also discussed.