Random forest is effective and efficient for detecting facial landmark from visual images, and has achieved the state-of-the-art performance, both in accuracy and speed, by regressing local binary features (LBF). This paper aims to increase the detection accuracy of random forest for facial landmarks and extends it to facial action estimation. First, probabilistic features are designed to overcome the weaknesses of LBF, e.g., feature sparseness and tracking jitter. Second, a deep architecture is introduced to random forest for enhancing the capacity of representation learning. Third, the initial detected facial landmarks are refined and 3D facial actions are estimated jointly by registering a deformable facial model to images based on an optimized iterative closest point framework. Experiments show that the proposed methods significantly outperform the state-of-the-art ones in terms of accuracy, as well as achieve the excellent tracking stability and real-time ability at about 60 fps on an ordinary PC.