This article presents an improved version of the coyote optimization algorithm (COA) that is more compatible with nature. In the proposed algorithm, fitness-distance balance (FDB) and Levy flight were used to determine the social tendency of coyote packs and to develop a more effective model imitating the birth of new coyotes. The balanced search performance, global exploration capability, and local exploitation ability of the COA algorithm were enhanced, and the premature convergence problem resolved using these two methods. The performance of the proposed Levy roulette FDB-COA (LRFDBCOA) was compared with 28 other meta-heuristic search (MHS) algorithms to verify its effectiveness on 90 benchmark test functions in different dimensions. The proposed LRFDBCOA and the COA ranked, respectively, the first and the ninth, according to nonparametric statistical results. The proposed algorithm was applied to solve the AC optimal power flow (ACOPF) problem incorporating thermal, wind, and combined solar-small hydro powered energy systems. This problem is described as a constrained, nonconvex, and complex power system optimization problem. The simulation results showed that the proposed algorithm exhibited a definite superiority over both the constrained and highly complex real-world engineering ACOPF problem and the unconstrained convex/nonconvex benchmark problems.