Infrared small target detection is one of the most important parts of infrared search and tracking (IRST) system. Generally, the small and dim target is of low signal-to-noise ratio and buried in the complicated background and heavy noise, which makes it extremely difficult to be detected with low false alarm rates. To solve this problem, we propose a small target detection method based on multiscale local contrast measure. Different from conventional methods, we novelly measure the local contrast from two aspects: local dissimilarity and local brightness difference. First, we present a new dissimilarity measure called the local energy factor (LEF) to describe the dissimilarity between the small targets and their surrounding backgrounds. Second, the feature of the brightness difference between the small targets and the backgrounds is utilized. Afterward, the local contrast is measured by taking both features of the above into account. Finally, an adaptive segmentation method is applied to extract the small targets from the backgrounds. Extensive experiments on real test data set demonstrate that our approach outperforms the state-of-the-art approaches.