Image cropping aims to enhance the aesthetic quality of a given image by searching for the good cropping views. One common routine is to score and rank the candidate views by the neural network. The network is expected to discriminate the subtle view-wise differences. However, the image-wise differences and the ambiguity in the annotations render difficulties in discriminating the view-wise differences. To focus on the view-wise differences, we propose a feature spliter to build image-wise and view-wise feature and evaluate the candidate views only based on the view-wise feature. Then, we propose the ranking gain loss that alleviates the ambiguity in annotations to amplify the view-wise differences. The remarkable improvement compared with prior arts on public benchmarks illustrates that the view-wise differences matter in cropping view recommendation.