In the paper, a modified coyote optimization algorithm (MCOA) is proposed for finding highly effective solutions for the optimal power flow (OPF) problem. In the OPF problem, total active power losses in all transmission lines and total electric generation cost of all available thermal units are considered to be reduced as much as possible meanwhile all constraints of transmission power systems such as generation and voltage limits of generators, generation limits of capacitors, secondary voltage limits of transformers, and limit of transmission lines are required to be exactly satisfied. MCOA is an improved version of the original coyote optimization algorithm (OCOA) with two modifications in two new solution generation techniques and one modification in the solution exchange technique. As compared to OCOA, the proposed MCOA has high contributions as follows: (i) finding more promising optimal solutions with a faster manner, (ii) shortening computation steps, and (iii) reaching higher success rate. Three IEEE transmission power networks are used for comparing MCOA with OCOA and other existing conventional methods, improved versions of these conventional methods, and hybrid methods. About the constraint handling ability, the success rate of MCOA is, respectively, 100%, 96%, and 52% meanwhile those of OCOA is, respectively, 88%, 74%, and 16%. About the obtained solutions, the improvement level of MCOA over OCOA can be up to 30.21% whereas the improvement level over other existing methods is up to 43.88%. Furthermore, these two methods are also executed for determining the best location of a photovoltaic system (PVS) with rated power of 2.0 MW in an IEEE 30-bus system. As a result, MCOA can reduce fuel cost and power loss by 0.5% and 24.36%. Therefore, MCOA can be recommended to be a powerful method for optimal power flow study on transmission power networks with considering the presence of renewable energies.