With the rapid development of depth data acquisition technology, the high precision depth becomes much easier to access in real-time by depth sensors, and the generated high dynamic range (HDR) depth is widely adopted to benefit the depth assistant applications. Accordingly, the HDR depth compression becomes essential for the efficient depth storage and transmission. In this paper, we introduce a layered compression framework for HDR depth to achieve efficient and low-complexity depth compression. To leverage the state-of-art 8-bit image/video encoders, the HDR depth is partitioned into two layers: most significant bit (MSB) layer and least significant bit (LSB) layer. For MSB layer, an error controllable pixel domain encoding scheme is proposed to guarantee the compatibility for existing 8-bit codec by controlling quantization errors added back to LSB layer. Meanwhile, the efficient major color extraction and adaptive quantization enhance the coding performance of MSB layer. For LSB layer, the layer data with limited dynamic range is compressed by normal 8-bit image/video based encoding scheme. The experimental results demonstrate that our coding scheme can achieve real-time depth compression with the satisfactory reconstruction quality. The encoding time is less than 31ms/frame and the decoding time is around 20ms/frame in average. Our compression scheme can be easily integrated into the real-time depth transmission system.