Along with the explosive growth of the Web, Web image search has become a more and more popular application which helps users digest the large amount of online visual information. Previous research mainly exploits visual information between images while rarely uses the text information surrounding the images on the Web pages. In this paper, we consider the relevance feedback as a machine learning problem. We proposed a novel relevance feedback framework for Web image search, which exploit both text and image modalities information with semi-supervised learning techniques. In each round of relevance feedbacks, the framework trains two classifiers for the two modalities by using the feedback information collected from the user. Then, it uses the unlabeled search result to improve these two classifiers. Finally, the ranked results list produced by image and text modality classifiers are combined to get the final rank. Experiments demonstrate the promise of the proposed framework.