The Lagrangian multiplier based rate-distortion optimization has been proved to be an effective way in hybrid video coding. In this paper, an advanced Lagrange multiplier selection method is presented. Based on Laplace distribution, the variance of transformed residuals is introduced into the rate and distortion models. Moreover, inspired by the ρ-domain method, the percentage of non-zeros among quantized residuals is also considered to refine the rate model. Thanks to the more accurate models, the proposed method is able to adaptively optimize the encoding process for videos with different properties. It outperforms the algorithm used in the reference software of H.264/AVC. According to the simulation, a gain up to 1.1dB was achieved.