This paper presents a new discriminative learning framework to associate the relationship between the objects and the words in an image and perform template matching scheme for complex association patterns. The problem is first formulated as a bipartite graph matching problem. Thereafter, structural support vector machine (SVM) is employed to obtain the optimal compatibility function to encode the association rules between the objects and the words. Moreover, an iterative inference procedure is developed to alternatively infer the association of visual objects and texts and the selection of the template model. Simulations show that the new method outperforms the existing competing counterparts.