Autonomous vehicles (AVs), as an emerging transportation tool, are widely discussed for improving traffic mobility. Particularly, when combining with prevalent shared transportation modes, i.e., car-sharing and ride-sharing, AVs show promising future. However, the efficiency of searching for an optimal match between shared autonomous vehicles and passengers has long become a critical problem. This research tackles this issue by developing a model which has better efficiency. The essential task is to dispatch vehicles to serve passengers, which is similar to vehicle routing problem. But unlike existing vehicle routing models, our method makes use of the similarity of passengers’ demand instead of directly matching vehicles and passengers. In this way, a cluster-based algorithm is proposed to find near-optimal solutions. Numerical experiments on large-size are performed to present the validity of the optimization model and efficiency of the proposed algorithm. Results indicate that the cluster-based algorithm could bring benefit to minimizing the number of vehicles and total travel distance.