Drones navigating in dark, GPS-denied and confined spaces can pose a difficult challenge due to, among other, the processing power required to maintain a high resolution map of the environment. This is specifically challenging when the drone has to fly through narrow spaces where a low resolution voxel representation can result in failure to find viable trajectories. Drawing inspiration from the Inspectrone Project, which deals with the inspection of large marine vessels for classification processes, in this paper we propose using a deep learning model to detect manholes relying only on a depth image. We investigate different sizes of networks in an attempt to provide an adequate accuracy while maintain a low computational load, suitable for drone implementation on a parallel processing co-processor. With an end goal to be able to accurately and robustly traverse the ballast tanks of the aforementioned vessels, we employ a temporal filter to increase the robustness and limit sensitivity to false positives by requiring multiple detections within a timeframe before the final location of the manhole is confirmed to be valid. Our results show that using deep learning on depth images is a feasible way to achieve a scene texture-agnostic solution for the detection the manholes. Our approach is successfully demonstrated by flying through a 1: 1 size standard manhole found on marine vessel.