Multimodal Language Independent App Classification Using Images and Text

Abstract

There are a number of methods for classification of mobile apps, but most of them rely on a fixed set of app categories and text descriptions associated with the apps. Often, one may need to classify apps into a different taxonomy and might have limited app usage data for the purpose. In this paper, we present an app classification system that uses object detection and recognition in images associated with apps, along with text based metadata of the apps, to generate a more accurate classification for a given app according to a given taxonomy. Our image based approach can, in principle, complement any existing text based approach for app classification. We train a fast RCNN to learn the coordinates of bounding boxes in an app image for effective object detection, as well as labels for the objects. We then use the detected objects in the app images in an ensemble with a text based system that uses a hierarchical supervised active learning pipeline based on uncertainty sampling for generating the training samples for a classifier. Using the ensemble, we are able to obtain better classification accuracy than if either of the text or image systems are used on their own.