Cross-media is the outstanding characteristics of the age of big data with large scale and complicated processing task. This article presents 5 issues and briefly summarizes the research progress of cross-media knowledge discovery. Furthermore, we propose a framework for cross-media semantic understanding which contains discriminative modeling, generative modeling and cognitive modeling. In cognitive modeling, a new model entitled CAM is proposed which is suitable for cross-media semantic understanding. Moreover, a Cross-Media Intelligent Retrieval System (CMIRS) will be illustrated. In the final, the research directions and problems encountered are presented.