Mobile Devices (MDs) support various delay/computation-intensive applications. Yet they only have limited battery energy and computing resources, thereby failing to totally run all applications. A mobile edge computing (MEC) paradigm has been proposed, and its servers are often deployed in both macro base stations (MBSs) and small base stations (SBSs). Thus, it is highly challenging to associate resource-limited MDs to them with high performance, and realize partial computation offloading among them for minimizing total energy consumption of an MEC system. This work formulates total energy consumption minimization as a constrained mixed integer non-linear program. To solve it, this work designs an improved meta-heuristic optimization algorithm called Particle swarm optimization based on Genetic Learning (PGL), which integrates strong local search capacity of a particle swarm optimizer, and genetic operations of a genetic algorithm. PGL jointly optimizes task offloading among MDs, SBSs and MBS, users’ connection to SBSs, MDs’ CPU speeds and transmission power, SBSs and MBS, and bandwidth allocation of available channels. Simulations with real-world data collected from Google cluster trace demonstrate that PGL significantly outperforms other existing methods in total energy consumption.