Esophageal cancer is the fastest rising type of cancer in the western world. Also, early neoplasia in Barrett's esophagus (BE) is difficult to detect for endoscopists and research has shown it is similarly complicated for Computer-Aided Detection (CAD) algorithms. For these reasons, further development of CAD algorithms for BE is essential for the wellbeing of patients. In this work we propose a patch-based deep learning algorithm for early neoplasia in BE, utilizing state-of-the-art deep learning techniques on a new prospective data set. The new algorithm yields not only a high detection score but also identifies the ideal biopsy location for the first time. We define specific novel metrics such as sweet-spot flag and softspot flag, to obtain well-defined computation of the biopsy location. Furthermore, we show that combining white light and blue laser imaging improves localization results by 8%.