Mobile edge computing (MEC) paradigm supports cloud-like computing capabilities at the edge of the network and offers low-latency services. Proxy servers of MEC with mobility and limited computing, e.g., flying unmanned aerial vehicles (UAVs) have emerged as competitors in providing services. This work considers a task offloading problem for an UAV-assisted MEC system and designs an integrated cloud-edge network with multiple mobile users (MUs) and layered UAVs to improve MEC with a network of UAVs. In our system, edge UAVs (EUAVs) and the cloud collaborate to provide caching and computing services for MUs. We consider static and dynamic applications that support task offloading. Our proposed approach minimizes the weighted cost of latency and energy consumption by jointly optimizing caching and offloading, deployment of EUAVs, and allocation of computation resources. Simultaneously, this work also considers UAVs’ caching and computation capacities while meeting MUs’ latency and energy constraints. Thus, a constrained mixed integer nonlinear program for a layered UAV-assisted hybrid cloud-edge system is formulated. To solve it, this work designs a hybrid metaheuristic algorithm named adaptive and genetic simulated annealing (SA)-based particle swarm optimization (AGSP). Experimental results with a real-life dataset verify that the AGSP’s system energy consumption and task latency are reduced by at least 7.4% and 8.46%, respectively, compared with the state-of-the-art algorithms, thus proving that AGSP greatly enhances the energy and latency of the system.