In this work, we present a supervised object segmentation algorithm for unconstrained video. Instead of arbitrarily picking a few frames for manual labeling, as in many existing supervised methods, the proposed method selects frames in a more reasonable manner, called supervision optimization. For this, we formulate a principled objective function by inferring the propagation error from appearance and motion clues. After this, we construct a multilevel segmentation model, which consists of low-level and high-level features. On the low level, image pixels are used for a more accurate estimation of motion and segmentation. On the high level, image segments are considered for a more semantic classification of the foreground and background. By integrating these in one segmentation graph, the result can be further improved by leveraging the knowledge from both levels. In experiments, the proposed approach is evaluated by different measures, and the results on a benchmark demonstrate the effectiveness in comparison with other state-of-the-art algorithms.