In the frequency division duplex mode, the limited reciprocity between uplink and downlink channels presents a significant challenge for the base station to obtain downlink channel state information (CSI). Existing solutions typically predict the downlink channel based on the estimated uplink CSI. However, the extra error caused by uplink channel estimation can be further accumulated into the downlink channel prediction, leading to inaccurate downlink channel prediction. Therefore, this letter proposes an end-to-end downlink channel prediction neural network directly based on uplink pilot, named E2ENet, which is immune to the extra error introduced by uplink channel estimation. Specifically, the channel we study is the complete time-frequency response within a time slot, featuring greater diversity that requires modeling. Therefore, we design a hybrid feature extraction module to solve this difficulty. Furthermore, we evolve E2ENet into a multi-output branch structure to simultaneously obtain uplink and downlink CSI to reduce storage overhead. Experimental results validate the effectiveness of our proposed solutions in improving the accuracy of downlink channel prediction and reducing storage overhead.