The next generation standard of coding, High Efficiency Video Coding (HEVC) aims to provide significantly improved compression performance in comparison with all existing video coding standards, such as MPEG-4, H.263, and H.264/AVC. The tool of rate-distortion optimization (RDO) mode decision has proven to be extremely important. Due to the increase in the number of intra prediction directions and large size of coding unit (CU), prediction unit (PU) and transform unit (TU), the number involved in RDO process rises dramatically, which is an extremely timing-and-computation-consuming process. In this paper, we propose a four-pixel-strip based ESAD and quantized TU coefficients based linear model to estimate the distortion and rate part of the RD cost function respectively. Experimental results show that the proposed RD cost function provides 85.8% area reduction and 1260% throughput improvement in hardware design, with negligible loss of bitrate and PSNR.