In this letter, we present a novel DCT-based local binary descriptor for the dense matching of multiple view stereo (MVS). Recently, although much progress has been made in the field of MVS, a key component of which, i.e., dense matching, is still a challenging task because it has two difficult issues: robust matching over non-salient regions (e.g., lines and textureless regions) and fast matching of a large number of pixels. To deal with these issues effectively, in the proposed dense descriptor, 2D DCT-based local features are utilized to achieve high discriminative power even for the non-salient regions. A binary representation is adopted to increase the matching performance as well as accelerate the matching speed via the Hamming distance. In addition, the discriminability of binarized vectors is further improved by a space-frequency pooling scheme. Through extensive experiments on the benchmark datasets for MVS, we demonstrate the superiority of the proposed descriptor over the state-of-the-art descriptors in terms of accuracy and efficiency.