Robotic urban search and rescue (USAR) is a challenging yet promising research area which has significant application potentials. This paper presents the development of a hierarchical reinforcement learning (HRL) based semi-autonomous controller for a rescue robot team working in cluttered and unstructured USAR environments. The HRL technique is introduced to address the robot exploration and victim identification problem in USAR environments, allowing rescue robots to learn from their previous experiences, their current surrounding settings, as well as the experience of other team members. Experiments conducted in a USAR-like environment verify the robustness of the proposed HRL-based semi-autonomous robot controller in unknown cluttered scenes.