Fluorescence imaging has been widely utilized in various clinical applications. As a functional imaging modality, NIR fluorescence imaging often does not offer sufficient structural details. Therefore, structural imaging such as color reflectance overlaid with fluorescence imaging represents a superior approach for surgical visualization. Image registration of color reflectance and NIR fluorescence is needed for accurate overlay. In this study, we have implemented a deep convolutional algorithm for feature-based fluorescence-to-color image registration. Software-hardware codesign was conducted. Several sets of experiments were performed on biological tissues to compare the performance of our algorithm and traditional methods. We have demonstrated the feasibility of deep convolutional feature-based fluorescence-to-color image registration. To our best knowledge, this is the first demonstration of deep learning-based image registration between fluorescence and color imageries.