This paper discusses the rate distortion optimization (RDO) issue in the scalable video coding. It is different from that in the non-scalable video coding because a scalable codec usually presents the decoded video within a certain range of the quality, frame rate and resolution. Therefore, this paper first analyzes the problem from a general model and gives a common description by the Lagrangian formula. Secondly, an algorithm is proposed to solve the above optimization problem at a specific scalable codec: the H.26L-based PFGS. At the same time, two analytic formulae are derived to ascertain the parameters /spl lambda/ in this algorithm. The experimental results show that the proposed algorithm significantly improves the coding efficiency of the H.26L-based PFGS. Further studies on this problem will be quite significant and challenging.