Recently, robot navigation methods using human instructions have been actively studied, including visual language navigation. Although language is one of the most promising forms of instruction, words often contain ambiguities. To complement this problem, we propose to use geometric instruction as a clue to the task goal. Specifically, in our proposed system, we assume that the robot receives a rough position of the target from human gesture. The robot adaptively estimates the reliability of this geometric instruction, and switches between exploration and instruction-following modes depending on the reliability value. We conducted evaluation of our method using a 3D simulation environment, and show that the task success rate and other metrics improve compared with the baseline methods.