We construct a quantum critical Otto engine that is powered by finite temperature baths. We show that the work output of the engine shows universal power law behavior that depends on the critical exponents of the working medium, as well as on the temperature of the cold bath. Furthermore, higher temperatures of the cold bath allows the engine to approach the limit of adiabatic operation for smaller values of the time period, while the corresponding power shows a maximum at an intermediate value of the cold bath temperature. These counterintuitive results stems from thermal excitations dominating the dynamics at higher temperatures.