In this paper, the maze navigation problem for the unmanned ground vehicle (UGV) is considered. A new maze navigation scheme with Reinforcement Learning (RL) is proposed to find the optimal path from the entrance to the exit for the UGV. First, the quadrotor with a camera at its bottom is used to capture the image data of a random maze in a 3D view, and an image processing approach is implemented to reconstruct the maze in a virtual platform. Then, a novel exploration algorithm, Q-Learning (λ) with improved e-greedy (iε-greedy), is proposed to find the shortest path in the reconstructed maze. The advantages of this paper are that the navigation scheme can provide an optimal path to the UGV without experiencing the maze cell by cell, and the proposed exploration algorithm can greatly reduce the randomness comparing to the traditional RL method. The effectiveness of both the proposed navigation scheme and the proposed exploration algorithm has been verified by simulations.