Autonomous exploration in unknown environments is a fundamental task of Unmanned Aerial Vehicles (UAVs). To choose exploration goals wisely, we propose an information-driven exploration strategy by applying the fast marching method to exploration of UAVs. A frontier point detection algorithm is designed to obtain Candidate Goals (CGs) by utilizing the structural characteristics of the octree-based map. With the sum of the information gain during the exploration journey as an evaluation indicator, we present a novel utility function to evaluate CGs by considering the trade-off between information gain and travel consumption. Given the effect of the environment on UAVs, UAVs are required to march aggressively in the Euclidean Symbol Distance Field (ESDF) to calculate the flight time, which is defined as the travel consumption. The uncertainty of the environment is minimized gradually by maximizing the utility function during each exploration journey. To take full advantage of the mobility of UAVs, we perform B-spline trajectory optimization and yaw angle planning based on the fast marching paths. We conduct sufficient comparison and evaluation experiments in simulation environments. Experimental results show the superiority of the proposed exploration strategy. The code related to the experiments will be published at https://github.com/BoLeiChen/fastmarching-exploration.