In this paper, we consider Heterogeneous Cloud Radio Access Network (H-CRAN) with non-uniform Device-to-Device (D2D) communication. Specifically, D2D transmission are only utilized outside a prescribed distance from any high power node (HPN). Based on the analytical framework provided in this paper, the average ergodic rate of a typical user equipment (UE) is characterized. Through appropriate definition of the exclusion area, the non-uniform D2D deployment can achieve performance improvement compared with uniform D2D deployment. In addition, to account for capacity constraint fronhaul of the remote radio heads (RRHs), we characterize the average traffic delivery latency experience by a typical UE when served by RRH as a quality of service (QoS) metric. Our results show that for lower fronthaul capacity regime, the non-uniform D2D deployment achieves lower average traffic delivery latency compared to both the uniform D2D deployment in H-CRAN and the pure H-CRAN scenarios.