When natural disasters strike, annotated images and texts flood the Internet, and rescue teams become overwhelmed to prioritize often scarce resources, while relying heavily on human input. In this paper, a novel multi-modal approach is proposed to automate crisis data analysis using machine learning. Our multi-modal two-stage framework relies on computationally inexpensive visual and semantic features to analyze Twitter data. Level I classification consists of training classifiers separately on semantic descriptors and combinations of visual features. These classifiers' decisions are aggregated to form a new feature vector to train the second set of classifiers in Level II classification. A home-grown dataset is gathered from Twitter to train the classifiers. Low-level visual features achieved an accuracy of 91.10% which increased to 92.43% when semantic attributes were incorporated. Applying such data science techniques on social media seems to motivate an updated folk statement "an ANNOTATED image is worth a thousand words".