Stochastic computing (SC) is a low-cost computation paradigm that can replace conventional binary arithmetic to provide a low hardware footprint with high scalability. However, since the SC bitstream length grows with the precision of the represented data, regardless of its lower power consumption, the convolutional SC-based neural networks may not be efficient in hardware area and energy. This work proposes a novel SC accelerator, PSC-Conv, to implement the convolutional layer using a new binary-interfaced stochastic computing architecture. PSC-Conv exploits kernel-wise parallelism in CNNs, reducing hardware footprint and energy consumption. Experimental re-sults show that the proposed implementation excels among several state-of-the-art SC-based implementations regarding area and power efficiency. We also compared the implementations of three modern CNNs, including LeNet-5, MobileNet, and ResNet-50. Experimental results demonstrate that, on average, PSC-Conv can achieve 5.02x speedup and 87.9% energy reduction compared with the binary implementation.