We study the challenging problem of following natural language instructions on a mobile manipulator robot. This task is challenging because it requires the robot to integrate the semantics of the unconstrained natural language instructions with the robot’s egocentric visual observations of the environment which are typically incomplete and noisy. To address these challenges, we propose a method that is able to use visible landmarks to more efficiently explore the environment in search of the objects described by the natural language instructions. Additionally, we propose using a pose adjustment policy during manipulation planning to help the robot recover from noisy visual observations. We show that this policy can be trained through experience with reinforcement learning as well as with human-in-the-loop feedback. We evaluate our approach on the popular ALFRED instruction following benchmark and show that these methods achieve state-of-the-art performance (35.41%) with a substantial (8.92% absolute) gap from prior work.