This paper describes an algorithm that efficiently groups line segments into perceptually salient contours in complex images. A measure of affinity between pairs of lines is used to guide group formation and limit the branching factor of the contour search procedure. The extracted contours are ranked, and presented as a contour hierarchy. Our algorithm is able to extract salient contours in the presence of texture, clutter, and repetitive or ambiguous image structure. We show experimental results on a complex line-set.