In this paper, we propose a novel graph-based salient object detection algorithm which exploits higher order potential to capture the cross-scale grouping cues instead of using multi-scale graph model or naive multi-scale fusion (i.e. individually compute a saliency result for each scale and then combine them). And, we investigate the importance of graph affinities in graph labeling. We take both local (spatial distribution) and nonlocal (feature distribution) priors into account and learn the pairwise similarity values in a semi-supervised manner, thereby obtaining a faithful graph affinity model. With the guidance of foreground and background seeds, salient object detection is formulated as a labeling inference problem. Extensive experiments on two large benchmark datasets demonstrate the proposed method performs well when against the state-of-the-art methods in terms of accuracy.