Optimal operation of cascade hydropower reservoirs is a complex high-dimensional engineering problem. Developing an appropriate model to solve such problems requires an efficient search method proportional to the dimensions of the problem. Accordingly, this research employed the new fitness-distance-balance (FDB) selection method in the moth swarm algorithm (MSA) to achieve promoted FDB-MSA with a high performance in solving complex large-scale problems. To ensure the efficiency of the developed algorithm, five benchmark functions of Shekel, Six-Hump Camel, McCormick, Goldstein-Price and Rosenbrock were used. Then, the FDB-MSA was used for optimization of hydropower generation of a real five-reservoir system along Karun River at Iran. This is the largest cascade reservoir system in Iran, which supplies more than 90% of the country’s hydropower demand. The results of the developed algorithm were compared with those of genetic algorithm (GA) and particle swarm optimization (PSO) algorithm. It was found that the FDB-MSA could successfully increase the hydropower generation by 59.5% (6724 GW) compared to the actual generation of energy over a 180-months operational period. The corresponding values for PSO and GA algorithms were 54.3% and 9.2% respectively. In addition, the results revealed the superiority of FDB-MSA to GA and PSO, so that, it demonstrated the smallest difference (3.41%) between nominal and optimal power generation compared to the PSO (6.58%) and GA (33.89%).