Semantic map is the foundation for mobile robots to understand its environment. By considering the semantic mapping problem as a semi-Markov process, a new hierarchical semi-Markov random field is proposed for this task. The proposed model can use multiple contextual information to label the places and objects in map and can partition the observations into spacial and semantic consistent sub-sequences each of which corresponding to a place. The proposed model is called coupled hidden semi-Markov conditional random fields (CHSM-CRFs). According to the structure of CHSM-CRFs, a piecewise learning algorithm and an approximating online inference algorithm based on Monte Carlo sampling are proposed for it. Experimental results with a mobile robot prove that the proposed method has high precision for labeling the places and objects in sematic mapping.