This article presents a multiobjective optimization method for a special cooling tooth design in an outer rotor permanent magnet starter generator (ORPMSG), taking into account both the electromagnetic and temperature performances. First, the key optimization parameters of the ORPMSG are determined based on analytical torque and temperature models. In addition, an optimization method considering the power density, efficiency, and temperature rise have been proposed and employed to obtain an optimal overall performance on the ORPMSG design. The comprehensive sensitivity method is conducted based on the random forest algorithm to improve model accuracy. Besides, the random black-hole algorithm is proposed to enhance the convergence rate and approximate Pareto optimal solution. Then, an optimal ORPMSG is obtained using the proposed multiobjective optimization. Besides, the corresponding electromagnetic and temperature characteristics of the optimal and initial machines are compared by finite element analysis to confirm the merits of the optimization method. Finally, a 25 kW prototype is manufactured and tested, which verifies a significant 45.1% current density and 3.2% efficiency improvement for the same condition compared to the traditional cooling method.