This paper proposed a novel Associative Memory model base on Self-organization Map(SOM), called MultiSOM. This model could learn associative relationships between data from different sources, mostly in different modality. However, data and relationships between them will not be entered into the network and trained directly. Instead, they should be trained each with a same semantic data and at last share one topological map. Cross-modally, this paper trains the MultiSOM model to learn associative memory between images and human voice of Chinese characters, with their meanings as sematic data, and the experiment results suggest that this MultiSOM model could learn the bidirectional associative relationship.