The fusion of multiple imaging modalities presents an important contribution to machine vision, but remains an ongoing challenge due to the limitations in traditional calibration methods that perform a single, global alignment. For depth and thermal imaging devices, sensor and lens intrinsics (FOV, resolution, etc.) may vary considerably, making per-pixel fusion accuracy difficult. In this paper, we present AccuFusion, a two-phase non-linear registration method to fuse multimodal images at a per-pixel level to obtain an efficient and accurate image registration. The two phases: the Coarse Fusion Network (CFN) and Refining Fusion Network (RFN), are designed to learn a robust image-space fusion that provides a non-linear mapping for accurate alignment. By employing the refinement process, we obtain per-pixel displacements to minimize local alignment errors and observe an increase of 18% in average accuracy over global registration.