This paper presents visual object tracking for handheld devices. Due to the low computation performance of handheld devices, the algorithms embedded on the devices are required to be performed with low computational consumption. The usability also should be considered at the same time. We adopted the drag-and-selection method to improve the user interactivity in the target object selection stage. To implement this method, we used flood-fill algorithm. In the main tracking process, we select mean-shift tracking as a main tracking algorithm. As a feature of mean-shift, we used color-based histogram backprojection. Additionally, the overall algorithm is modified for improving the speed of the tracking process. We validated the algorithm on the handheld device with low computation power and tested in several real situations.