Camera-based remote photoplethysmography (rPPG) enables low-cost, non-contact cardiovascular activity monitoring. However, applying rPPG to practical use has some limitations caused from the artifacts by illuminance changes. During watching a video in a dark room, for example, watching a TV at night without illuminance, there is a high correlation between the brightness changes of a video and the illuminance variation on the skin of the viewer's face. In this study, we propose an artifact reduction method in rPPG, which is caused by the variation of the illuminance. The method subtracts the artifacts from the raw facial rPPG signal by applying multi-order curve fitting between the illuminance information from the facial rPPG signal and the brightness information from a video. On average, the results showed that signal-to-noise ratio (SNR) increased from -11.74 to -4.19 dB and from -15.27 to 7.99 dB for low-dynamic-brightness and high-dynamic-brightness video, respectively. In addition, the root-mean-square-error (RMSE) of estimated heart rate decreased from 11.00 to 1.82 bpm and from 9.88 to 4.65 bpm for the videos, respectively.