Millions of people in the world suffer from vision impairment or even vision loss. Guide sticks and dogs have been deployed to lead them around various obstacles. However, both of them are not capable of interacting with human users who normally rely on conceptual knowledge or semantic contents of the environment. This paper first builds a 3D semantic indoor environment map with an RGB-D sensor. Then, the map is used for room recognition during the revisits based on appearance by applying a convolutional neural network. Representative objects extracted from the semantic map are used to diagnose and eliminate errors during room recognition. The proposed method result in a 97.8% accuracy even with lighting condition and small object location changes.