The performance of an FPGA based CNN accelerator is determined by both parallelism and frequency, however, most prior works optimize the parallelism in the RTL design and resolve the frequency after the synthesis. This paper presents a design space exploration method for the pipeline implementation of the deep CNN models, which concurrently optimizes parallelism and frequency to achieve a comprehensive optimization on throughput. In addition to the quantitative modeling on parallelism, the maximum achievable system frequency under various parallelism is explored to leverage the PVT-margins in real-life scenarios and is adopted to guide the design space exploration for further performance boost. A case study of the AlexNet model is implemented using the proposed method on the Altera DE5a-Net board. The experimental results demonstrate that our method can achieve the throughput up to 906.25GOP/s, which gains 1.39× improvement compared to state-of-the-art RTL optimization methods.