Scale variation, perspective distortion and severe occlusion are three main problems that affect the accuracy of crowd counting. Existing methods either adopt attention mechanism or perspective values to address these problems, but without considering them as a whole. In this paper, we advocate introducing perspective information across different density distributions to facilitate crowd estimation, and propose a novel model named Perspective-aware Density Regression Network (PDRNet) for crowd counting. Unlike previous works, PDRNet is a bilateral structure with different focuses on the features of perspective and crowd density, and it includes the information interaction module (IIM) and the similarity comparison module (SCM) to enhance the perspective-density interaction in a cross-reference manner. Specifically, IIM achieves the mutual feature guidance by swapping the global information between different branches, and SCM performs feature comparison to refine the prediction of foreground regions. With extensive experiments and competitive performances on four widely used datasets, we demonstrate the effectiveness of the proposed network.