Mobility load balancing (MLB) aims to solve the problem of uneven resource utilization in cellular networks. Since network dynamics are usually complicated and non-stationary, conventional model-based MLB methods fail to cover all scenarios of cellular networks. On the other hand, deep reinforcement learning (DRL) can provide a flexible framework to learn to distribute cell load evenly without explicit modeling of the underlying network dynamics. In this paper, we introduce a novel decentralized DRL-based MLB method where each cell has a DRL agent to learn its handover parameters and antenna tilt angle. As the number of cells increases, the decentralized framework is more computationally efficient than its centralized counterpart by dividing the action space. Furthermore, our designed decentralized DRL architecture only requires readily known information defined in existing cellular standards, and it can achieve a more balanced cell load distribution than the centralized DRL one by using individual reward functions. To provide realistic performance evaluation, a network simulator is introduced strictly following the Third Generation Partnership Project (3GPP) specifications. Furthermore, field data is used to construct the underlying cellular environment. Extensive evaluations have been conducted to demonstrate the fact that the introduced decentralized DRL-based MLB method can achieve a more balanced cell load distribution and a better performance of edge users than the state-of-the-art MLB methods.