Self-driving vehicles are being increasingly deployed in the wild. One of the most important next hurdles for autonomous driving is how such vehicles will optimally interact with one another and with their surroundings. In this paper, we consider the lane changing problem that is fundamental to road-bound multi-vehicle systems, and approach it through a combination of deep reinforcement learning (DRL) and game theory. We introduce a proactive-passive lane changing framework and formulate the lane changing problem as a Markov game between the proactive and passive vehicles. Based on different approaches to carry out DRL to solve the Markov game, we propose an asynchronous lane changing scheme as in a single-agent RL setting and a synchronous cooperative lane changing scheme that takes into consideration the adaptive behavior of the other vehicle in a vehicle's decision. Experimental results show that the synchronous scheme can effectively create and find proper merging moment after sufficient training. The framework and solution developed here demonstrate the potential of using reinforcement learning to solve multi-agent autonomous vehicle tasks such as the lane changing as they are formulated as Markov games.