Blood oxygen saturation (SpO$_2$) is an important indicator forpulmonary and respiratory functionalities. Clinical findings on COVID-19 show that many patients had dangerously low blood oxygen levels not long before conditions worsened. It is therefore recommended, especially for the vulnerable population, to regularly monitor the blood oxygen level for precaution. Recent works have investigated how ubiquitous smartphone cameras can be used to infer SpO$_2$. Most of these works are contact-based, requiring users to cover a phone’s camera and its nearby light source with a finger to capture reemitted light from the illuminated tissue. Contact-based methods may lead to skin irritation and sanitary concerns, especially during a pandemic. In this paper, we propose a noncontact method for SpO$_2$ monitoring using hand videos acquired by smartphones. Considering the optical broadband nature of the red (R), green (G), and blue (B) color channels of the smartphone cameras, we exploit all three channels of RGB sensing to distill the SpO$_2$ information beyond the traditional ratio-of-ratios (RoR) method that uses only two wavelengths. To further facilitate an accurate SpO$_2$ prediction, we design adaptive narrow bandpass filters based on accurately estimated heart rate to obtain the most cardiac-related AC component for each color channel. Experimental results show that our proposed blood oxygen estimation method can reach a mean absolute error of 1.26% when a pulse oximeter is used as a reference, outperforming the traditional RoR method by 25%.