A number of recent studies of information diffusion in social media, both empirical and theoretical, have been inspired by viral propagation models derived from epidemiology. These studies model propagation of memes, i.e., pieces of information, between users in a social network similarly to the way diseases spread in human society. Naturally, many of these studies emphasize social exposure, i.e., the number of friends or acquaintances of a user that have exposed a meme to her, as the primary metric for understanding, predicting, and controlling information diffusion. Intuitively, one would expect a meme to spread in a social network selectively, i.e., amongst the people who are interested in the meme. However, the importance of the alignment between the topicality of a meme and the topical interests of the potential adopters and influencers in the network has been less explored in the literature. In this paper, we quantify the impact of the topical alignment between memes and users on their adoption. Our analysis, using empirical data about two different types of memes, i.e., hashtags and URLs spreading through the Twitter social media platform, finds that topical alignment between memes and users is as crucial as the social exposure in understanding and predicting meme adoptions. Our results emphasize the need to look beyond social network-based viral propagation models and develop microscopic models of information diffusion that account for interests of users and topicality of information.