As the explosive growth of mobile data traffic, there is a general trend toward the decentralization of mobile networks. However, existing gateway (GW) relocation strategy is tailored for centralized mobile operator networks. Resources of the Evolved Packet Core (EPC) for decentralized networks will not be utilized effectively without rethinking gateway selection schemes. Furthermore, user experience may be deteriorated severely as well. In this paper, we propose a Cost-Effective Optimal Gateway Selection (CEOGS) algorithm for decentralized mobile networks. By considering the factors of both data plane and control plane, we will select the gateway that has the maximum grey association degree with the deal optimal GW. The performances of the proposed CEOGS algorithm are evaluated through simulations by comparing current standardized techniques. Encouraging results are obtained, demonstrating the benefits of the proposed CEOGS scheme over the other alternatives.