Edge detection is a fundamental problem in computer vision community. In this paper, we propose a novel concept for edge detection called Structural Edge. The Structural edges include occluding contours of objects as well as orientation discontinuities in surfaces that define the 3D structure of objects and their environments. This contrasts the semantic edge which is only the boundary between semantic areas. While existing edge detection methods focus on either semantic boundaries or low-level gradients, we focus on the structural edge. To achieve that, in this paper, we propose the structural edge dataset along with a benchmark. The structural edge dataset contains 600 images of natural indoor and outdoor scenes. The structural edges are labeled manually and validated by eye-tracking data from 10 participants with overall 20 trials. Later, we use the dataset to benchmark the existing edge detection methods. We benchmark both the learning based and non-learning based methods and draw the conclusion that existing methods cannot fully solve the structural edge detection. We encourage new research to exploit the proposed task.