In this paper we provide novel insights about the performance and design of popular pairwise tests-based local binary descriptors with the aim of answering the question: How many bits are needed for matching local binary descriptors? We use the interpretation of binary descriptors as a Locality Sensitive Hashing (LSH) scheme for approximating Kendall's tau rank distance between image patches. Based on this understanding we compare local binary descriptors in terms of the number of bits that are required to achieve a certain performance in feature-based matching problems. Furthermore, we introduce a calibration method to automatically determine a suitable number of bits required in an image matching scenario. We provide a performance analysis in image matching and structure from motion benchmarks, showing calibration results in visual odometry and object recognition problems. Our results show that excellent performance can be achieved using a small fraction of the total number of bits from the whole descriptor, speeding-up matching and reducing storage requirements.