Visual discomfort is one of the most frequent complaints of the viewers when watching stereo images or videos. It has been the subject of considerable research in relation to stereo display system. To predict the effects induced by three-dimensional contents on visual discomfort, a new objective visual comfort assessment (VCA) method of stereo image is proposed based on scene mode classification. Scene modes are defined and real scenes are classified into ten kinds of scene modes according to the properties of two aspects. These aspects include the crossed / uncrossed disparity type of foreground object (FGO) and background region (BGR), and the other is the whether locate on zone of comfortable viewing of FGO and BGR in scene. In the process of mode classification and selection, disparity map is first utilized to segment stereo image into FGOs and BGRs adaptively. Then, some features, including disparity angle of both the FGOs and BGRs, and width angle of FGOs, are utilized to build an objective VCA models in various scene modes. The experimental results show that the proposed method performs higher assessment accuracy than some state-of-the-art methods.