We propose two very effective high-level binary-class features to enhance model-based skin color detection. First we find that the log likelihood ratio of the testing data between skin and non-skin RGB models can be a good discriminative feature. We also find that namely the background-foreground correlation provides another complementary feature compared to the conventional low-level RGB feature. Further improvement can be accomplished by Bayesian model adaptation and feature fusion. By jointly considering both schemes of Bayesian model adaptation and feature fusion, we attain the best system performance. Experimental results show that the proposed joint framework improves the 68% to 84% baseline F₁ scores to as high as almost 90% in a wide range of lighting conditions.