Prostate cancer (PCa) is the fifth leading cause of death world-wide. In spite of the urgency for a timely and accurate diagnostic, the current PCa diagnostic pathway suffers from over-diagnosis of indolent lesions and under-diagnosis of highly invasive ones. The advent of deep learning (DL) techniques has enabled automatic and accurate computer-assisted systems that rival human performance. However, current approaches for PCa diagnostic are heavily reliant on T2w axial MRI, which suffer from low out-of-plane resolution. Sagittal and coronal MRI scans are usually acquired by default along with the axial one but are generally ignored by DL classification algorithms. We propose a multi-stream approach to accommodate sagittal, coronal and axial planes and improve the performance of PCa lesion classification. We evaluate our method on a publicly available dataset and demonstrate that it provides better results when compared with a single-plane approach over a range of different DL architectures.