The knowledge graph, a networked structure designed to organize the vast and heterogeneous knowledge existing in the real world, has gained widespread adoption as a background knowledge base for intelligent systems. Nevertheless, the incompleteness of knowledge graphs has been widely recognized as a significant challenge in their development and application. Recently, multi-hop knowledge graph reasoning has been an attractive method for completing a knowledge graph. The multi-hop knowledge graph reasoning based on the reinforcement learning (RL) framework has achieved promising performance in terms of interpretability and scalability. An RL agent automatically reasons over a KG under the guidance of a policy network. When faced with a query, obtaining the approximate range of the answer first and then delving into individual options is a more efficient approach compared to traversing all candidate answers. However, existing RL-based methods have two limitations. First, they lack the ability to filter candidate decisions, making it challenging to handle entities with a large number of neighbors. Second, they fail to consider the intrinsic correlations between entities. To address these limitations, we propose a novel hierarchical knowledge graph reasoning approach HiKGR, which leverages the concept information of entities. Specifically, HiKGR reconstructs the previous action space in RL into a concept space and an instance space, enabling two policies to alternate reasoning at the concept level and the instance level. Furthermore, we propose hierarchical reward functions for the two-level policies to achieve joint optimization. The hierarchical reasoning approach we propose is capable of selecting more reasonable candidate decisions and optimizing the decision space. Experimental results reveal that HiKGR significantly outperforms existing RL-based methods and drastically reduces the action space size.