Recently, location-based services (LBSs) have been increasingly popular for people to experience new possibilities, for example, personalized point-of-interest (POI) recommendations that leverage on the overlapping of user trajectories to recommend POI collaboratively. POI recommendation is yet challenging as it suffers from the problems known for the conventional recommendation tasks such as data sparsity and cold start, and to a much greater extent. In the literature, most of the related works apply collaborate filtering to POI recommendation while overlooking the personalized time-variant human behavioral tendency. In this article, we put forward a fourth-order tensor factorization-based ranking methodology to recommend users their interested locations by considering their time-varying behavioral trends while capturing their long-term preferences and short-term preferences simultaneously. We also propose to categorize the locations to alleviate data sparsity and cold-start issues, and accordingly new POIs that users have not visited can thus be bubbled up during the category ranking process. The tensor factorization is carefully studied to prune the irrelevant factors to the ranking results to achieve efficient POI recommendations. The experimental results validate the efficacy of our proposed mechanism, which outperforms the state-of-the-art approaches significantly.